2024-09-11T08:06:41.491 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T08:06:41.505 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T08:06:41.577 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274 branch: wip-guits-quincy-2024-09-10-2008-quincy description: orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} email: gabrioux@ibm.com first_in_suite: false job_id: '7901274' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-quincy-2024-09-10-2008-quincy ceph: conf: global: mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-quincy-2024-09-10-2008-quincy sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 2282 sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-quincy-2024-09-10-2008-quincy suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d0e6828a2016d48cf25ad84064e50742bb1c39b9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBN82PVGp9MC63bTCYdFHMmoIT4Q4JHpzes40Di8fTDL5DVD7BZe4BI7gweUB8L4vlwCKbaMRX4IlMtuePJQlyE= smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3h6sCV3+dxNj3EFfvNfhCEYHflIeto+6bLeDo3CnmmmcVoT8licyRPwONGx/VO3KFLVh/bfuzF2lf+fnS/YsI= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - "set -ex\nfor f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x\n\ do\n echo \"rotating key for $f\"\n K=$(ceph auth get-key $f)\n NK=\"\ $K\"\n ceph orch daemon rotate-key $f\n while [ \"$K\" == \"$NK\" ]; do\n\ \ sleep 5\n NK=$(ceph auth get-key $f)\n done\ndone\n" teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_06:53:43 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T08:06:41.577 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d0e6828a2016d48cf25ad84064e50742bb1c39b9/qa; will attempt to use it 2024-09-11T08:06:41.578 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d0e6828a2016d48cf25ad84064e50742bb1c39b9/qa/tasks 2024-09-11T08:06:41.578 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T08:06:41.579 INFO:teuthology.task.internal:Checking packages... 2024-09-11T08:06:41.600 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd0e6828a2016d48cf25ad84064e50742bb1c39b9' 2024-09-11T08:06:41.600 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T08:06:41.600 INFO:teuthology.packaging:ref: None 2024-09-11T08:06:41.600 INFO:teuthology.packaging:tag: None 2024-09-11T08:06:41.600 INFO:teuthology.packaging:branch: wip-guits-quincy-2024-09-10-2008-quincy 2024-09-11T08:06:41.600 INFO:teuthology.packaging:sha1: d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:06:41.600 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-quincy-2024-09-10-2008-quincy 2024-09-11T08:06:41.752 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1623.gd0e6828a 2024-09-11T08:06:41.754 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T08:06:41.761 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T08:06:41.761 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T08:06:41.802 INFO:teuthology.task.internal:Saving configuration 2024-09-11T08:06:41.813 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T08:06:41.887 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T08:06:41.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 08:01:44.882469', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBN82PVGp9MC63bTCYdFHMmoIT4Q4JHpzes40Di8fTDL5DVD7BZe4BI7gweUB8L4vlwCKbaMRX4IlMtuePJQlyE='} 2024-09-11T08:06:41.932 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 08:01:44.880697', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3h6sCV3+dxNj3EFfvNfhCEYHflIeto+6bLeDo3CnmmmcVoT8licyRPwONGx/VO3KFLVh/bfuzF2lf+fnS/YsI='} 2024-09-11T08:06:41.933 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T08:06:41.941 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-11T08:06:41.941 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-11T08:06:41.942 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T08:06:42.029 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f78e448fac0>, signals=[15]) 2024-09-11T08:06:42.029 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T08:06:42.035 INFO:teuthology.task.internal:Opening connections... 2024-09-11T08:06:42.035 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-09-11T08:06:42.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:06:42.119 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-09-11T08:06:42.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:06:42.193 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T08:06:42.199 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-09-11T08:06:42.217 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-09-11T08:06:42.217 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-09-11T08:06:42.276 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-09-11T08:06:42.277 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T08:06:42.278 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T08:06:42.278 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T08:06:42.278 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-09-11T08:06:42.297 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-09-11T08:06:42.316 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-09-11T08:06:42.316 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-09-11T08:06:42.374 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2024-09-11T08:06:42.374 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="9" 2024-09-11T08:06:42.374 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2024-09-11T08:06:42.374 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2024-09-11T08:06:42.374 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="9" 2024-09-11T08:06:42.374 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el9" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:LOGO="fedora-logo-icon" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T08:06:42.375 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T08:06:42.375 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-09-11T08:06:42.405 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T08:06:42.415 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T08:06:42.457 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T08:06:42.458 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-09-11T08:06:42.460 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-09-11T08:06:42.477 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T08:06:42.500 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T08:06:42.500 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-09-11T08:06:42.517 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-09-11T08:06:42.622 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T08:06:42.676 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T08:06:42.709 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T08:06:42.709 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T08:06:42.709 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T08:06:42.710 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T08:06:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:06:42.727 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-09-11T08:06:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:06:42.728 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-09-11T08:06:42.784 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-503.el9.x86_64 2024-09-11T08:06:42.785 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-503.el9.x86_64 2024-09-11T08:06:42.785 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-09-11T08:06:42.786 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-503.el9.x86_64 2024-09-11T08:06:42.786 INFO:teuthology.task.kernel:Running kernel on smithi139: 5.14.0-503.el9.x86_64 2024-09-11T08:06:42.786 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2024-09-11T08:06:45.243 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-11T08:06:46.342 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-11T08:06:46.927 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 35 MB/s | 20 MB 00:00 2024-09-11T08:06:50.050 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 20 MB 00:02 2024-09-11T08:06:51.420 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.5 MB 00:00 2024-09-11T08:06:52.878 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-11T08:06:53.799 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-11T08:06:55.536 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-11T08:06:57.870 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-11T08:06:58.782 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-11T08:06:59.179 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-09-11T08:07:01.177 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T08:07:01.178 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T08:07:01.263 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:07:01.272 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:07:01.272 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:07:01.273 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2024-09-11T08:07:01.274 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:07:01.274 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 79 M 2024-09-11T08:07:01.274 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 126 M 2024-09-11T08:07:01.274 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:07:01.448 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 176 kB/s | 23 kB 00:00 2024-09-11T08:07:02.831 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-11T08:07:03.323 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 15 MB/s | 29 MB 00:01 2024-09-11T08:07:03.631 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 15 MB/s | 35 MB 00:02 2024-09-11T08:07:03.632 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:07:03.632 INFO:teuthology.orchestra.run.smithi139.stdout:Total 33 MB/s | 79 MB 00:02 2024-09-11T08:07:04.196 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-11T08:07:04.314 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:07:04.686 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:07:04.686 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:07:05.647 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:07:05.647 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:07:06.203 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T08:07:06.203 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T08:07:06.288 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:07:06.295 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:07:06.295 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-11T08:07:06.295 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:07:06.295 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:07:06.295 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-09-11T08:07:06.296 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:07:06.297 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 79 M 2024-09-11T08:07:06.297 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 126 M 2024-09-11T08:07:06.297 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:07:06.873 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 76 kB/s | 23 kB 00:00 2024-09-11T08:07:08.099 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:07:08.806 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T08:07:08.889 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T08:07:11.703 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T08:07:11.826 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T08:07:16.678 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T08:07:16.718 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:07:21.380 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.1 MB/s | 16 MB 00:14 2024-09-11T08:07:21.585 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:07:41.464 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 845 kB/s | 29 MB 00:34 2024-09-11T08:07:47.405 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 866 kB/s | 35 MB 00:40 2024-09-11T08:07:47.405 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:07:47.406 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.9 MB/s | 79 MB 00:41 2024-09-11T08:07:48.089 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:07:48.443 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:07:48.443 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:07:49.380 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:07:49.380 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:07:51.825 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:07:52.521 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T08:07:52.605 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T08:07:55.374 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T08:07:55.507 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T08:07:59.475 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:07:59.515 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:07:59.868 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:07:59.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T08:07:59.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T08:07:59.868 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T08:08:00.333 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T08:08:00.440 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:08:00.688 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:08:00.688 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:08:00.689 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:08:00.689 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T08:08:00.689 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T08:08:00.689 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T08:08:00.689 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T08:08:00.689 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:08:00.689 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:08:01.047 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T08:08:01.144 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T08:08:01.145 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T08:08:01.145 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T08:08:01.145 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T08:08:01.145 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T08:08:01.145 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-11T08:08:01.145 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T08:08:01.145 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T08:08:01.203 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T08:08:01.204 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T08:08:01.204 DEBUG:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.06 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 82.el9 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Group : Unspecified 2024-09-11T08:08:01.236 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 8274670 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : builder@centos.org 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T08:08:01.237 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T08:08:01.238 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2024-09-11T08:08:01.238 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:08:01.238 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T08:08:01.238 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T08:08:01.239 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 5.14.0-505.el9.x86_64 2024-09-11T08:08:01.239 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T08:08:01.939 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2024-09-11T08:08:03.269 INFO:teuthology.orchestra.run.smithi139.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T08:08:03.308 INFO:teuthology.orchestra.run.smithi139.stderr:done 2024-09-11T08:08:03.310 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-09-11T08:08:03.326 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.kiECpaSBZT 2024-09-11T08:08:03.326 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kiECpaSBZT 2024-09-11T08:08:03.393 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.kiECpaSBZT 2024-09-11T08:08:03.536 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.kiECpaSBZT is 6KB 2024-09-11T08:08:03.557 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.kiECpaSBZT 2024-09-11T08:08:03.572 DEBUG:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T08:08:03.635 INFO:teuthology.orchestra.run.smithi139.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T08:08:03.635 INFO:teuthology.orchestra.run.smithi139.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T08:08:03.635 INFO:teuthology.orchestra.run.smithi139.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T08:08:03.635 INFO:teuthology.orchestra.run.smithi139.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T08:08:03.636 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T08:08:03.804 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2024-09-11T08:08:05.273 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:08:33.807 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T08:08:33.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-11T08:08:33.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:08:43.090 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:08:43.129 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:08:43.493 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:08:43.493 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T08:08:43.493 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T08:08:43.493 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T08:08:44.346 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T08:08:44.346 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:08:44.347 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:08:44.347 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T08:08:44.347 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T08:08:44.347 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T08:08:44.347 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T08:08:44.347 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:08:44.347 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:08:44.708 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T08:08:44.794 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T08:08:44.794 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T08:08:44.794 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T08:08:44.795 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T08:08:44.795 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T08:08:44.795 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T08:08:44.795 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T08:08:44.795 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T08:08:44.854 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T08:08:44.854 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T08:08:44.854 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-09-11T08:08:44.886 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-09-11T08:08:44.886 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 82.el9 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8274670 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T08:08:44.887 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:08:44.888 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T08:08:44.889 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T08:08:44.889 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-505.el9.x86_64 2024-09-11T08:08:44.890 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T08:08:45.565 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-09-11T08:08:47.001 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T08:08:47.038 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-09-11T08:08:47.041 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-09-11T08:08:47.097 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.VcZoUj0wJZ 2024-09-11T08:08:47.097 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VcZoUj0wJZ 2024-09-11T08:08:47.179 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.VcZoUj0wJZ 2024-09-11T08:08:47.316 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.VcZoUj0wJZ is 6KB 2024-09-11T08:08:47.365 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.VcZoUj0wJZ 2024-09-11T08:08:47.380 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T08:08:47.470 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T08:08:47.470 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T08:08:47.470 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T08:08:47.470 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T08:08:47.471 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T08:08:47.574 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-09-11T08:08:52.285 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-11T08:09:01.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-11T08:09:01.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:09:04.349 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-11T08:09:16.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-11T08:09:16.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:09:17.578 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T08:09:17.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-11T08:09:17.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:09:48.140 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-09-11T08:09:48.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-11T08:09:48.489 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-11T08:09:48.489 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-09-11T08:09:48.504 INFO:teuthology.orchestra.run.smithi139.stdout:5.14.0-505.el9.x86_64 2024-09-11T08:09:48.505 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T08:09:48.505 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T08:09:48.505 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T08:09:49.506 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T08:09:49.507 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T08:09:49.633 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2024-09-11T08:09:49.663 DEBUG:teuthology.parallel:result is None 2024-09-11T08:10:17.582 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T08:10:26.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-11T08:10:26.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:10:27.330 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-09-11T08:10:27.728 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-11T08:10:27.728 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T08:10:27.728 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-09-11T08:10:27.743 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-505.el9.x86_64 2024-09-11T08:10:27.743 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T08:10:27.743 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T08:10:27.743 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T08:10:28.744 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T08:10:28.744 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T08:10:28.829 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-09-11T08:10:28.857 DEBUG:teuthology.parallel:result is None 2024-09-11T08:10:28.857 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T08:10:28.864 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T08:10:28.865 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T08:10:28.867 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T08:10:28.886 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T08:10:28.911 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T08:10:28.970 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T08:10:28.970 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T08:10:28.972 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T08:10:29.000 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T08:10:29.072 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T08:10:29.073 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T08:10:29.075 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T08:10:29.116 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:10:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:10:29.127 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:10:29.139 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:10:29.141 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T08:10:29.189 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T08:10:29.189 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T08:10:29.192 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T08:10:29.221 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T08:10:29.286 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T08:10:29.286 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T08:10:29.288 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T08:10:29.309 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T08:10:29.389 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T08:10:29.437 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T08:10:29.524 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T08:10:29.562 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:10:29.562 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T08:10:29.632 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T08:10:29.681 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T08:10:29.727 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T08:10:29.814 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T08:10:29.854 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:10:29.854 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T08:10:29.921 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-09-11T08:10:29.923 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-09-11T08:10:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T08:10:30.059 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T08:10:30.359 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T08:10:30.367 INFO:teuthology.task.internal:Starting timer... 2024-09-11T08:10:30.367 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T08:10:30.405 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T08:10:30.435 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-11T08:10:30.435 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-09-11T08:10:30.478 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-09-11T08:10:30.480 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2024-09-11T08:10:30.531 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2024-09-11T08:10:30.533 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T08:10:30.533 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-09-11T08:10:30.554 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-09-11T08:10:30.554 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2024-09-11T08:10:30.576 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2024-09-11T08:10:30.576 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive'} 2024-09-11T08:10:30.577 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T08:10:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.623 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T08:10:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:10:30.650 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T08:10:30.650 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-09-11T08:10:30.700 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2024-09-11T08:10:30.738 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T08:10:30.747 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T08:10:30.763 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T08:10:30.764 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi139.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T08:14:18.297 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com')] 2024-09-11T08:14:18.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-11T08:14:18.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:14:18.377 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-09-11T08:14:18.455 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-11T08:14:18.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-11T08:14:18.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:14:18.530 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-09-11T08:14:18.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-11T08:14:18.620 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T08:14:18.631 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T08:14:18.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T08:14:18.632 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:14:18.634 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T08:14:18.634 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:14:18.659 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T08:14:18.673 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T08:14:18.688 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T08:14:18.697 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-09-11T08:14:18.702 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T08:14:18.709 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-09-11T08:14:18.722 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T08:14:18.726 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2024-09-11T08:14:18.736 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T08:14:18.737 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2024-09-11T08:14:18.751 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T08:14:18.765 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T08:14:18.794 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-09-11T08:14:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T08:14:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-09-11T08:14:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.826 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-09-11T08:14:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T08:14:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-09-11T08:14:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.830 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:14:18.831 INFO:teuthology.run_tasks:Running task pexec... 2024-09-11T08:14:18.841 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-11T08:14:18.841 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi110.front.sepia.ceph.com 2024-09-11T08:14:18.841 DEBUG:teuthology.orchestra.run.smithi110:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-11T08:14:18.842 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi139.front.sepia.ceph.com 2024-09-11T08:14:18.842 DEBUG:teuthology.orchestra.run.smithi139:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-11T08:14:19.192 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:14:19.197 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:19.197 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-11T08:14:19.197 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:19.197 INFO:teuthology.orchestra.run.smithi110.stdout:Removing: 2024-09-11T08:14:19.197 INFO:teuthology.orchestra.run.smithi110.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-11T08:14:19.197 INFO:teuthology.orchestra.run.smithi110.stdout:Removing unused dependencies: 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout:Remove 2 Packages 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout:Freed space: 5.5 M 2024-09-11T08:14:19.198 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:14:19.202 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:14:19.202 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:14:19.220 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout:Removing: 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout:Removing unused dependencies: 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:19.236 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:14:19.237 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:19.237 INFO:teuthology.orchestra.run.smithi139.stdout:Remove 2 Packages 2024-09-11T08:14:19.237 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:19.237 INFO:teuthology.orchestra.run.smithi139.stdout:Freed space: 5.5 M 2024-09-11T08:14:19.237 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:14:19.240 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:14:19.240 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:14:19.262 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:14:19.262 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:14:19.307 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:14:19.307 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:14:19.393 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:14:19.448 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:14:19.481 INFO:teuthology.orchestra.run.smithi110.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-11T08:14:19.533 INFO:teuthology.orchestra.run.smithi139.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-11T08:14:19.552 INFO:teuthology.orchestra.run.smithi110.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T08:14:19.588 INFO:teuthology.orchestra.run.smithi139.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T08:14:20.029 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T08:14:20.029 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-11T08:14:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-11T08:14:20.101 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-11T08:14:20.252 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-11T08:14:20.252 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:20.252 INFO:teuthology.orchestra.run.smithi110.stdout:Removed: 2024-09-11T08:14:20.252 INFO:teuthology.orchestra.run.smithi110.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T08:14:20.252 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:20.252 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:14:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-11T08:14:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:20.374 INFO:teuthology.orchestra.run.smithi139.stdout:Removed: 2024-09-11T08:14:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T08:14:20.374 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:20.374 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:14:21.056 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:44 ago on Wed 11 Sep 2024 08:13:36 AM UTC. 2024-09-11T08:14:21.179 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Sep 2024 08:13:28 AM UTC. 2024-09-11T08:14:21.201 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:14:21.201 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:14:21.202 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:21.203 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-09-11T08:14:21.203 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:21.211 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2024-09-11T08:14:21.211 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 9.2 M 2024-09-11T08:14:21.211 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:14:21.321 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:14:21.322 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:21.322 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-09-11T08:14:21.322 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:21.322 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2024-09-11T08:14:21.323 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:21.336 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 2.0 M 2024-09-11T08:14:21.336 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 9.2 M 2024-09-11T08:14:21.336 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:14:21.599 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 418 kB/s | 44 kB 00:00 2024-09-11T08:14:21.604 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 306 kB/s | 44 kB 00:00 2024-09-11T08:14:21.624 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 805 kB/s | 106 kB 00:00 2024-09-11T08:14:21.662 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 520 kB/s | 106 kB 00:00 2024-09-11T08:14:21.674 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 953 kB/s | 72 kB 00:00 2024-09-11T08:14:21.700 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.1 MB/s | 84 kB 00:00 2024-09-11T08:14:21.721 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 613 kB/s | 72 kB 00:00 2024-09-11T08:14:21.741 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.5 MB/s | 906 kB 00:00 2024-09-11T08:14:21.771 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.8 MB/s | 906 kB 00:00 2024-09-11T08:14:21.797 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): python3-kmod-0.9-32.el9.x86_64.rpm 629 kB/s | 84 kB 00:00 2024-09-11T08:14:21.809 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.1 MB/s | 837 kB 00:00 2024-09-11T08:14:21.809 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:14:21.809 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2024-09-11T08:14:21.883 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:14:21.896 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:14:21.896 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:14:21.897 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.7 MB/s | 837 kB 00:00 2024-09-11T08:14:21.897 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:14:21.897 INFO:teuthology.orchestra.run.smithi139.stdout:Total 3.6 MB/s | 2.0 MB 00:00 2024-09-11T08:14:21.969 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:14:21.982 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:14:21.982 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:14:22.205 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:14:22.205 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:14:22.280 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:14:22.280 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:14:22.620 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:14:22.697 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:14:22.719 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-11T08:14:22.754 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-11T08:14:22.783 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-11T08:14:22.790 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-11T08:14:22.825 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-11T08:14:22.854 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-11T08:14:22.989 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-11T08:14:23.022 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T08:14:23.058 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-11T08:14:23.085 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T08:14:23.452 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T08:14:23.492 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T08:14:23.528 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-11T08:14:23.557 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T08:14:24.435 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T08:14:24.436 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-11T08:14:24.436 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-11T08:14:24.436 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-11T08:14:24.436 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-11T08:14:24.436 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-11T08:14:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-11T08:14:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-11T08:14:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-11T08:14:24.566 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-11T08:14:24.567 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-11T08:14:24.567 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:24.759 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:24.901 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:14:25.079 DEBUG:teuthology.parallel:result is None 2024-09-11T08:14:25.234 DEBUG:teuthology.parallel:result is None 2024-09-11T08:14:25.234 INFO:teuthology.run_tasks:Running task install... 2024-09-11T08:14:25.243 DEBUG:teuthology.task.install:project ceph 2024-09-11T08:14:25.243 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9'}} 2024-09-11T08:14:25.244 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9'} 2024-09-11T08:14:25.244 INFO:teuthology.task.install:Using flavor: default 2024-09-11T08:14:25.249 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T08:14:25.249 INFO:teuthology.task.install:extra packages: [] 2024-09-11T08:14:25.249 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9', 'tag': None, 'wait_for_package': False} 2024-09-11T08:14:25.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:14:25.252 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9', 'tag': None, 'wait_for_package': False} 2024-09-11T08:14:25.252 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:14:25.399 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/ 2024-09-11T08:14:25.400 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1623.gd0e6828a 2024-09-11T08:14:25.404 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/ 2024-09-11T08:14:25.404 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1623.gd0e6828a 2024-09-11T08:14:25.509 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T08:14:25.509 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:14:25.509 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T08:14:25.517 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-10-2008-quincy/d0e6828a2016d48cf25ad84064e50742bb1c39b9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T08:14:25.517 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:14:25.517 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T08:14:25.538 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T08:14:25.538 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e6828a2016d48cf25ad84064e50742bb1c39b9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T08:14:25.547 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T08:14:25.548 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d0e6828a2016d48cf25ad84064e50742bb1c39b9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T08:14:25.605 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T08:14:25.625 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T08:14:25.682 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T08:14:25.701 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T08:14:25.758 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2024-09-11T08:14:25.760 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2024-09-11T08:14:25.766 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-09-11T08:14:25.768 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-09-11T08:14:25.977 INFO:teuthology.orchestra.run.smithi139.stdout:45 files removed 2024-09-11T08:14:25.986 INFO:teuthology.orchestra.run.smithi110.stdout:45 files removed 2024-09-11T08:14:26.004 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw 2024-09-11T08:14:26.013 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw 2024-09-11T08:14:26.561 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 287 kB/s | 77 kB 00:00 2024-09-11T08:14:26.573 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 280 kB/s | 77 kB 00:00 2024-09-11T08:14:26.836 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 47 kB/s | 11 kB 00:00 2024-09-11T08:14:26.849 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 46 kB/s | 11 kB 00:00 2024-09-11T08:14:27.095 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 8.0 kB/s | 1.8 kB 00:00 2024-09-11T08:14:27.116 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-11T08:14:27.603 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-11T08:14:28.475 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.2 MB 00:01 2024-09-11T08:14:29.813 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-11T08:14:34.000 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-11T08:14:37.346 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.5 MB 00:03 2024-09-11T08:14:39.451 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-11T08:14:39.614 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-11T08:14:40.551 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-11T08:14:40.885 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 19 kB 00:00 2024-09-11T08:14:41.832 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-11T08:14:45.915 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-11T08:14:47.280 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-09-11T08:14:48.293 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:14:48.295 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:14:48.295 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:48.295 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:14:48.295 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 14 M 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.1 M 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 5.0 M 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 23 M 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 636 k 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T08:14:48.296 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 457 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 4.9 M 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 45 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 120 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 156 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 320 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 294 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 99 k 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T08:14:48.297 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:48.298 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:14:48.298 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:14:48.298 INFO:teuthology.orchestra.run.smithi139.stdout:Install 22 Packages 2024-09-11T08:14:48.298 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 2 Packages 2024-09-11T08:14:48.298 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:48.298 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 58 M 2024-09-11T08:14:48.298 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:14:49.008 INFO:teuthology.orchestra.run.smithi139.stdout:(1/24): ceph-base-17.2.7-1623.gd0e6828a.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-11T08:14:49.083 INFO:teuthology.orchestra.run.smithi139.stdout:(2/24): ceph-selinux-17.2.7-1623.gd0e6828a.el9. 335 kB/s | 25 kB 00:00 2024-09-11T08:14:49.183 INFO:teuthology.orchestra.run.smithi139.stdout:(3/24): libcephfs2-17.2.7-1623.gd0e6828a.el9.x8 6.2 MB/s | 636 kB 00:00 2024-09-11T08:14:49.275 INFO:teuthology.orchestra.run.smithi139.stdout:(4/24): libradosstriper1-17.2.7-1623.gd0e6828a. 4.9 MB/s | 457 kB 00:00 2024-09-11T08:14:49.634 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:14:49.636 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:49.636 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 14 M 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.1 M 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 5.0 M 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 23 M 2024-09-11T08:14:49.637 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 636 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 457 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 4.9 M 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T08:14:49.638 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 45 k 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 120 k 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 156 k 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 320 k 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 294 k 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 99 k 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:14:49.639 INFO:teuthology.orchestra.run.smithi110.stdout:Install 22 Packages 2024-09-11T08:14:49.640 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2024-09-11T08:14:49.640 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:49.640 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 58 M 2024-09-11T08:14:49.640 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:14:49.734 INFO:teuthology.orchestra.run.smithi139.stdout:(5/24): ceph-common-17.2.7-1623.gd0e6828a.el9.x 19 MB/s | 23 MB 00:01 2024-09-11T08:14:49.876 INFO:teuthology.orchestra.run.smithi139.stdout:(6/24): ceph-radosgw-17.2.7-1623.gd0e6828a.el9. 11 MB/s | 14 MB 00:01 2024-09-11T08:14:49.901 INFO:teuthology.orchestra.run.smithi139.stdout:(7/24): python3-ceph-argparse-17.2.7-1623.gd0e6 271 kB/s | 45 kB 00:00 2024-09-11T08:14:49.926 INFO:teuthology.orchestra.run.smithi139.stdout:(8/24): python3-ceph-common-17.2.7-1623.gd0e682 2.3 MB/s | 120 kB 00:00 2024-09-11T08:14:49.951 INFO:teuthology.orchestra.run.smithi139.stdout:(9/24): python3-cephfs-17.2.7-1623.gd0e6828a.el 3.0 MB/s | 156 kB 00:00 2024-09-11T08:14:50.027 INFO:teuthology.orchestra.run.smithi139.stdout:(10/24): librgw2-17.2.7-1623.gd0e6828a.el9.x86_ 6.5 MB/s | 4.9 MB 00:00 2024-09-11T08:14:50.060 INFO:teuthology.orchestra.run.smithi139.stdout:(11/24): python3-rados-17.2.7-1623.gd0e6828a.el 2.3 MB/s | 320 kB 00:00 2024-09-11T08:14:50.085 INFO:teuthology.orchestra.run.smithi139.stdout:(12/24): python3-rbd-17.2.7-1623.gd0e6828a.el9. 2.1 MB/s | 294 kB 00:00 2024-09-11T08:14:50.119 INFO:teuthology.orchestra.run.smithi139.stdout:(13/24): python3-rgw-17.2.7-1623.gd0e6828a.el9. 1.1 MB/s | 99 kB 00:00 2024-09-11T08:14:50.247 INFO:teuthology.orchestra.run.smithi110.stdout:(1/24): ceph-base-17.2.7-1623.gd0e6828a.el9.x86 9.6 MB/s | 5.0 MB 00:00 2024-09-11T08:14:50.306 INFO:teuthology.orchestra.run.smithi110.stdout:(2/24): ceph-selinux-17.2.7-1623.gd0e6828a.el9. 433 kB/s | 25 kB 00:00 2024-09-11T08:14:50.406 INFO:teuthology.orchestra.run.smithi110.stdout:(3/24): libcephfs2-17.2.7-1623.gd0e6828a.el9.x8 6.2 MB/s | 636 kB 00:00 2024-09-11T08:14:50.590 INFO:teuthology.orchestra.run.smithi110.stdout:(4/24): ceph-radosgw-17.2.7-1623.gd0e6828a.el9. 17 MB/s | 14 MB 00:00 2024-09-11T08:14:50.623 INFO:teuthology.orchestra.run.smithi110.stdout:(5/24): libradosstriper1-17.2.7-1623.gd0e6828a. 2.1 MB/s | 457 kB 00:00 2024-09-11T08:14:50.678 INFO:teuthology.orchestra.run.smithi139.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 76 kB/s | 45 kB 00:00 2024-09-11T08:14:50.690 INFO:teuthology.orchestra.run.smithi110.stdout:(6/24): python3-ceph-argparse-17.2.7-1623.gd0e6 678 kB/s | 45 kB 00:00 2024-09-11T08:14:50.786 INFO:teuthology.orchestra.run.smithi139.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 221 kB/s | 160 kB 00:00 2024-09-11T08:14:50.790 INFO:teuthology.orchestra.run.smithi110.stdout:(7/24): librgw2-17.2.7-1623.gd0e6828a.el9.x86_6 25 MB/s | 4.9 MB 00:00 2024-09-11T08:14:50.816 INFO:teuthology.orchestra.run.smithi110.stdout:(8/24): python3-ceph-common-17.2.7-1623.gd0e682 960 kB/s | 120 kB 00:00 2024-09-11T08:14:50.841 INFO:teuthology.orchestra.run.smithi110.stdout:(9/24): python3-cephfs-17.2.7-1623.gd0e6828a.el 3.0 MB/s | 156 kB 00:00 2024-09-11T08:14:50.866 INFO:teuthology.orchestra.run.smithi110.stdout:(10/24): python3-rados-17.2.7-1623.gd0e6828a.el 6.2 MB/s | 320 kB 00:00 2024-09-11T08:14:50.891 INFO:teuthology.orchestra.run.smithi110.stdout:(11/24): python3-rbd-17.2.7-1623.gd0e6828a.el9. 5.7 MB/s | 294 kB 00:00 2024-09-11T08:14:50.917 INFO:teuthology.orchestra.run.smithi110.stdout:(12/24): python3-rgw-17.2.7-1623.gd0e6828a.el9. 1.9 MB/s | 99 kB 00:00 2024-09-11T08:14:50.995 INFO:teuthology.orchestra.run.smithi139.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 756 kB/s | 662 kB 00:00 2024-09-11T08:14:51.062 INFO:teuthology.orchestra.run.smithi139.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.6 MB/s | 309 kB 00:00 2024-09-11T08:14:51.087 INFO:teuthology.orchestra.run.smithi139.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-11T08:14:51.113 INFO:teuthology.orchestra.run.smithi139.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-11T08:14:51.167 INFO:teuthology.orchestra.run.smithi110.stdout:(13/24): ceph-common-17.2.7-1623.gd0e6828a.el9. 16 MB/s | 23 MB 00:01 2024-09-11T08:14:51.171 INFO:teuthology.orchestra.run.smithi139.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-09-11T08:14:51.192 INFO:teuthology.orchestra.run.smithi110.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 164 kB/s | 45 kB 00:00 2024-09-11T08:14:51.284 INFO:teuthology.orchestra.run.smithi110.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 408 kB/s | 160 kB 00:00 2024-09-11T08:14:51.335 INFO:teuthology.orchestra.run.smithi110.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-11T08:14:51.360 INFO:teuthology.orchestra.run.smithi110.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 562 kB/s | 42 kB 00:00 2024-09-11T08:14:51.372 INFO:teuthology.orchestra.run.smithi139.stdout:(21/24): librados2-17.2.7-1623.gd0e6828a.el9.x8 15 MB/s | 3.1 MB 00:00 2024-09-11T08:14:51.388 INFO:teuthology.orchestra.run.smithi139.stdout:(22/24): python3-prettytable-0.7.2-27.el9.noarc 70 kB/s | 42 kB 00:00 2024-09-11T08:14:51.419 INFO:teuthology.orchestra.run.smithi110.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-09-11T08:14:51.444 INFO:teuthology.orchestra.run.smithi110.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2024-09-11T08:14:51.469 INFO:teuthology.orchestra.run.smithi110.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2024-09-11T08:14:51.489 INFO:teuthology.orchestra.run.smithi139.stdout:(23/24): librbd1-17.2.7-1623.gd0e6828a.el9.x86_ 25 MB/s | 2.9 MB 00:00 2024-09-11T08:14:51.503 INFO:teuthology.orchestra.run.smithi110.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.9 MB/s | 662 kB 00:00 2024-09-11T08:14:51.539 INFO:teuthology.orchestra.run.smithi139.stdout:(24/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 340 kB/s | 292 kB 00:00 2024-09-11T08:14:51.543 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:14:51.544 INFO:teuthology.orchestra.run.smithi139.stdout:Total 18 MB/s | 58 MB 00:03 2024-09-11T08:14:51.553 INFO:teuthology.orchestra.run.smithi110.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-11T08:14:51.605 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:14:51.639 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:14:51.639 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:14:51.745 INFO:teuthology.orchestra.run.smithi110.stdout:(23/24): librados2-17.2.7-1623.gd0e6828a.el9.x8 11 MB/s | 3.1 MB 00:00 2024-09-11T08:14:51.820 INFO:teuthology.orchestra.run.smithi110.stdout:(24/24): librbd1-17.2.7-1623.gd0e6828a.el9.x86_ 9.3 MB/s | 2.9 MB 00:00 2024-09-11T08:14:51.825 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:14:51.825 INFO:teuthology.orchestra.run.smithi110.stdout:Total 27 MB/s | 58 MB 00:02 2024-09-11T08:14:51.886 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:14:51.920 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:14:51.920 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:14:52.102 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:14:52.102 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:14:52.390 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:14:52.391 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:14:53.300 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:14:53.661 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:14:53.700 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-11T08:14:53.749 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T08:14:53.842 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T08:14:53.918 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T08:14:53.942 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T08:14:54.019 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-11T08:14:54.031 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T08:14:54.076 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-11T08:14:54.105 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-11T08:14:54.151 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T08:14:54.248 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T08:14:54.326 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T08:14:54.366 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T08:14:54.445 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T08:14:54.498 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-11T08:14:54.537 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-11T08:14:54.557 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-11T08:14:54.595 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T08:14:54.658 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T08:14:54.694 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-11T08:14:54.738 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 10/26 2024-09-11T08:14:54.774 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T08:14:54.836 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T08:14:54.864 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T08:14:54.930 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T08:14:54.974 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-11T08:14:54.981 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-11T08:14:55.010 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T08:14:55.022 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-11T08:14:55.056 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-11T08:14:55.082 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 8/26 2024-09-11T08:14:55.110 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-11T08:14:55.154 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 10/26 2024-09-11T08:14:55.190 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T08:14:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T08:14:55.280 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T08:14:55.329 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 13/26 2024-09-11T08:14:55.357 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-11T08:14:55.370 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-11T08:14:55.401 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T08:14:55.422 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-11T08:14:55.454 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-11T08:14:55.491 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T08:14:55.573 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 19/26 2024-09-11T08:14:55.603 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 20/26 2024-09-11T08:14:55.756 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-11T08:14:55.792 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T08:14:55.883 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 18/26 2024-09-11T08:14:55.956 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 19/26 2024-09-11T08:14:55.987 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 20/26 2024-09-11T08:14:57.969 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T08:14:57.999 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T08:14:58.429 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T08:14:58.471 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T08:14:58.510 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T08:14:58.544 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T08:14:58.646 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T08:14:58.646 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T08:14:58.646 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T08:14:58.646 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:14:58.678 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T08:14:59.003 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 21/26 2024-09-11T08:14:59.041 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T08:14:59.138 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 22/26 2024-09-11T08:14:59.138 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T08:14:59.138 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T08:14:59.138 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:14:59.175 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T08:15:08.343 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T08:15:08.343 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2024-09-11T08:15:08.344 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2024-09-11T08:15:08.344 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2024-09-11T08:15:08.344 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2024-09-11T08:15:08.344 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2024-09-11T08:15:08.344 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /root 2024-09-11T08:15:08.344 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2024-09-11T08:15:08.344 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:08.925 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-09-11T08:15:08.926 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:09.694 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T08:15:09.756 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T08:15:09.757 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:09.757 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T08:15:09.757 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:15:09.757 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:15:09.757 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:09.758 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T08:15:09.847 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T08:15:09.847 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T08:15:10.253 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T08:15:10.322 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 24/26 2024-09-11T08:15:10.323 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:10.323 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T08:15:10.323 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:15:10.323 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:15:10.323 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:10.324 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T08:15:10.422 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-11T08:15:10.422 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 6/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 7/26 2024-09-11T08:15:11.103 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 8/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 9/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 10/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 13/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-11T08:15:11.104 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-11T08:15:11.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-11T08:15:11.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-11T08:15:11.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-11T08:15:11.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-11T08:15:11.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T08:15:11.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-11T08:15:11.105 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 25/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_6 6/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 7/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9. 8/26 2024-09-11T08:15:11.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x8 9/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 10/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 11/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 12/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 13/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-11T08:15:11.632 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-11T08:15:11.633 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-11T08:15:11.633 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 23/26 2024-09-11T08:15:11.633 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-11T08:15:11.633 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 25/26 2024-09-11T08:15:11.946 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T08:15:11.946 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:11.946 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2024-09-11T08:15:11.946 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.946 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.946 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.947 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:11.948 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:12.294 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-test 2024-09-11T08:15:12.654 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-11T08:15:12.654 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:12.654 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-09-11T08:15:12.654 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.654 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.654 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T08:15:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T08:15:12.656 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:12.657 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:12.867 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:27 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:12.996 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test 2024-09-11T08:15:13.000 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 48 M 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 163 k 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:13.001 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:15:13.002 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:13.002 INFO:teuthology.orchestra.run.smithi139.stdout:Install 4 Packages 2024-09-11T08:15:13.002 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:13.002 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 48 M 2024-09-11T08:15:13.002 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 193 M 2024-09-11T08:15:13.002 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:15:13.241 INFO:teuthology.orchestra.run.smithi139.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-11T08:15:13.283 INFO:teuthology.orchestra.run.smithi139.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-11T08:15:13.333 INFO:teuthology.orchestra.run.smithi139.stdout:(3/4): libcephsqlite-17.2.7-1623.gd0e6828a.el9. 722 kB/s | 163 kB 00:00 2024-09-11T08:15:13.558 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:26 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:13.689 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 48 M 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 163 k 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T08:15:13.692 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T08:15:13.693 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:13.693 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:15:13.693 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:13.693 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-09-11T08:15:13.693 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:13.693 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-09-11T08:15:13.694 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 193 M 2024-09-11T08:15:13.694 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:15:14.285 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): libcephsqlite-17.2.7-1623.gd0e6828a.el9. 489 kB/s | 163 kB 00:00 2024-09-11T08:15:14.684 INFO:teuthology.orchestra.run.smithi139.stdout:(4/4): ceph-test-17.2.7-1623.gd0e6828a.el9.x86_ 30 MB/s | 48 MB 00:01 2024-09-11T08:15:14.686 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:14.686 INFO:teuthology.orchestra.run.smithi139.stdout:Total 29 MB/s | 48 MB 00:01 2024-09-11T08:15:14.714 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:15:14.727 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 393 kB/s | 304 kB 00:00 2024-09-11T08:15:14.733 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:15:14.733 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:15:14.952 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-09-11T08:15:15.099 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:15:15.100 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:15:15.617 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:15:15.627 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): ceph-test-17.2.7-1623.gd0e6828a.el9.x86_ 29 MB/s | 48 MB 00:01 2024-09-11T08:15:15.629 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:15.629 INFO:teuthology.orchestra.run.smithi110.stdout:Total 25 MB/s | 48 MB 00:01 2024-09-11T08:15:15.657 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:15:15.677 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:15:15.677 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:15:15.714 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T08:15:15.757 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T08:15:15.785 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T08:15:16.043 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:15:16.044 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:15:16.575 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:15:16.662 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T08:15:16.710 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T08:15:16.738 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T08:15:20.193 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T08:15:20.267 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T08:15:21.180 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/4 2024-09-11T08:15:21.262 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T08:15:22.518 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T08:15:22.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/4 2024-09-11T08:15:22.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/4 2024-09-11T08:15:22.518 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T08:15:22.825 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T08:15:22.825 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:22.825 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:15:22.825 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:22.825 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:22.826 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T08:15:22.826 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T08:15:22.826 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:22.826 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:23.121 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph 2024-09-11T08:15:23.507 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/4 2024-09-11T08:15:23.507 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/4 2024-09-11T08:15:23.507 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/4 2024-09-11T08:15:23.507 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T08:15:23.689 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:38 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:23.816 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T08:15:23.816 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:23.816 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:15:23.816 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:23.816 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:23.817 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T08:15:23.817 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T08:15:23.817 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:23.817 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:23.824 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:23.826 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 6.5 k 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.1 M 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 1.4 M 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 240 k 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.9 M 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 15 M 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T08:15:23.827 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T08:15:23.828 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T08:15:23.829 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout:Install 34 Packages 2024-09-11T08:15:23.830 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:23.831 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 27 M 2024-09-11T08:15:23.831 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 96 M 2024-09-11T08:15:23.831 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:15:24.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph 2024-09-11T08:15:24.698 INFO:teuthology.orchestra.run.smithi139.stdout:(1/34): ceph-17.2.7-1623.gd0e6828a.el9.x86_64.r 45 kB/s | 6.5 kB 00:00 2024-09-11T08:15:24.703 INFO:teuthology.orchestra.run.smithi139.stdout:(2/34): ceph-mgr-17.2.7-1623.gd0e6828a.el9.x86_ 4.8 MB/s | 1.4 MB 00:00 2024-09-11T08:15:24.771 INFO:teuthology.orchestra.run.smithi139.stdout:(3/34): ceph-mds-17.2.7-1623.gd0e6828a.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-11T08:15:24.863 INFO:teuthology.orchestra.run.smithi139.stdout:(4/34): ceph-mon-17.2.7-1623.gd0e6828a.el9.x86_ 13 MB/s | 3.9 MB 00:00 2024-09-11T08:15:24.888 INFO:teuthology.orchestra.run.smithi139.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1623.gd0e6 2.0 MB/s | 240 kB 00:00 2024-09-11T08:15:25.013 INFO:teuthology.orchestra.run.smithi139.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-11T08:15:25.063 INFO:teuthology.orchestra.run.smithi139.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-11T08:15:25.247 INFO:teuthology.orchestra.run.smithi139.stdout:(8/34): ceph-osd-17.2.7-1623.gd0e6828a.el9.x86_ 28 MB/s | 15 MB 00:00 2024-09-11T08:15:25.255 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:38 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:25.272 INFO:teuthology.orchestra.run.smithi139.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 648 kB/s | 135 kB 00:00 2024-09-11T08:15:25.305 INFO:teuthology.orchestra.run.smithi139.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-09-11T08:15:25.331 INFO:teuthology.orchestra.run.smithi139.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-11T08:15:25.381 INFO:teuthology.orchestra.run.smithi139.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 932 kB/s | 125 kB 00:00 2024-09-11T08:15:25.396 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:25.398 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:25.398 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:15:25.398 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:25.398 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:15:25.398 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 6.5 k 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.1 M 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 1.4 M 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 240 k 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 3.9 M 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 15 M 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T08:15:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T08:15:25.400 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout:Install 34 Packages 2024-09-11T08:15:25.401 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:25.402 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 27 M 2024-09-11T08:15:25.402 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 96 M 2024-09-11T08:15:25.402 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:15:25.615 INFO:teuthology.orchestra.run.smithi139.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 122 kB/s | 35 kB 00:00 2024-09-11T08:15:25.815 INFO:teuthology.orchestra.run.smithi139.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 255 kB/s | 51 kB 00:00 2024-09-11T08:15:25.840 INFO:teuthology.orchestra.run.smithi139.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 168 kB/s | 77 kB 00:00 2024-09-11T08:15:25.857 INFO:teuthology.orchestra.run.smithi110.stdout:(1/34): ceph-17.2.7-1623.gd0e6828a.el9.x86_64.r 41 kB/s | 6.5 kB 00:00 2024-09-11T08:15:25.874 INFO:teuthology.orchestra.run.smithi139.stdout:(16/34): python3-mako-1.1.4-6.el9.noarch.rpm 303 kB/s | 172 kB 00:00 2024-09-11T08:15:25.908 INFO:teuthology.orchestra.run.smithi139.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 648 kB/s | 43 kB 00:00 2024-09-11T08:15:25.932 INFO:teuthology.orchestra.run.smithi139.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 354 kB/s | 42 kB 00:00 2024-09-11T08:15:25.966 INFO:teuthology.orchestra.run.smithi139.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-09-11T08:15:25.991 INFO:teuthology.orchestra.run.smithi139.stdout:(20/34): python3-cherrypy-18.6.1-2.el9.noarch.r 4.3 MB/s | 358 kB 00:00 2024-09-11T08:15:26.017 INFO:teuthology.orchestra.run.smithi139.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-09-11T08:15:26.024 INFO:teuthology.orchestra.run.smithi110.stdout:(2/34): ceph-mgr-17.2.7-1623.gd0e6828a.el9.x86_ 4.3 MB/s | 1.4 MB 00:00 2024-09-11T08:15:26.042 INFO:teuthology.orchestra.run.smithi139.stdout:(22/34): python3-jaraco-classes-3.2.1-5.el9.noa 239 kB/s | 18 kB 00:00 2024-09-11T08:15:26.067 INFO:teuthology.orchestra.run.smithi139.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-09-11T08:15:26.091 INFO:teuthology.orchestra.run.smithi110.stdout:(3/34): ceph-mds-17.2.7-1623.gd0e6828a.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2024-09-11T08:15:26.092 INFO:teuthology.orchestra.run.smithi139.stdout:(24/34): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-11T08:15:26.118 INFO:teuthology.orchestra.run.smithi139.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-11T08:15:26.143 INFO:teuthology.orchestra.run.smithi139.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-09-11T08:15:26.150 INFO:teuthology.orchestra.run.smithi110.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1623.gd0e6 4.0 MB/s | 240 kB 00:00 2024-09-11T08:15:26.168 INFO:teuthology.orchestra.run.smithi139.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-11T08:15:26.194 INFO:teuthology.orchestra.run.smithi139.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-11T08:15:26.219 INFO:teuthology.orchestra.run.smithi139.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-11T08:15:26.244 INFO:teuthology.orchestra.run.smithi139.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-09-11T08:15:26.270 INFO:teuthology.orchestra.run.smithi139.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-11T08:15:26.303 INFO:teuthology.orchestra.run.smithi139.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 611 kB/s | 20 kB 00:00 2024-09-11T08:15:26.328 INFO:teuthology.orchestra.run.smithi139.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-11T08:15:26.353 INFO:teuthology.orchestra.run.smithi139.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-11T08:15:26.357 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:26.357 INFO:teuthology.orchestra.run.smithi139.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-11T08:15:26.482 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:15:26.517 INFO:teuthology.orchestra.run.smithi110.stdout:(5/34): ceph-osd-17.2.7-1623.gd0e6828a.el9.x86_ 31 MB/s | 15 MB 00:00 2024-09-11T08:15:26.518 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:15:26.518 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:15:26.575 INFO:teuthology.orchestra.run.smithi110.stdout:(6/34): ceph-mon-17.2.7-1623.gd0e6828a.el9.x86_ 5.5 MB/s | 3.9 MB 00:00 2024-09-11T08:15:26.894 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:15:26.894 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:15:27.092 INFO:teuthology.orchestra.run.smithi110.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 269 kB/s | 253 kB 00:00 2024-09-11T08:15:27.242 INFO:teuthology.orchestra.run.smithi110.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 160 kB/s | 106 kB 00:00 2024-09-11T08:15:27.467 INFO:teuthology.orchestra.run.smithi110.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 1.3 MB/s | 1.2 MB 00:00 2024-09-11T08:15:27.809 INFO:teuthology.orchestra.run.smithi110.stdout:(10/34): python3-pycparser-2.20-6.el9.noarch.rp 189 kB/s | 135 kB 00:00 2024-09-11T08:15:27.951 INFO:teuthology.orchestra.run.smithi110.stdout:(11/34): python3-requests-2.25.1-8.el9.noarch.r 176 kB/s | 125 kB 00:00 2024-09-11T08:15:27.985 INFO:teuthology.orchestra.run.smithi110.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 985 kB/s | 172 kB 00:00 2024-09-11T08:15:28.018 INFO:teuthology.orchestra.run.smithi110.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 523 kB/s | 35 kB 00:00 2024-09-11T08:15:28.094 INFO:teuthology.orchestra.run.smithi110.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-09-11T08:15:28.119 INFO:teuthology.orchestra.run.smithi110.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 576 kB/s | 77 kB 00:00 2024-09-11T08:15:28.148 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:15:28.161 INFO:teuthology.orchestra.run.smithi110.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 623 kB/s | 42 kB 00:00 2024-09-11T08:15:28.187 INFO:teuthology.orchestra.run.smithi110.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 650 kB/s | 43 kB 00:00 2024-09-11T08:15:28.220 INFO:teuthology.orchestra.run.smithi110.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 14 MB/s | 358 kB 00:00 2024-09-11T08:15:28.222 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-11T08:15:28.245 INFO:teuthology.orchestra.run.smithi110.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 466 kB/s | 11 kB 00:00 2024-09-11T08:15:28.255 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-11T08:15:28.271 INFO:teuthology.orchestra.run.smithi110.stdout:(20/34): python3-urllib3-1.26.5-6.el9.noarch.rp 270 kB/s | 215 kB 00:00 2024-09-11T08:15:28.296 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-11T08:15:28.296 INFO:teuthology.orchestra.run.smithi110.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-11T08:15:28.330 INFO:teuthology.orchestra.run.smithi110.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 760 kB/s | 19 kB 00:00 2024-09-11T08:15:28.355 INFO:teuthology.orchestra.run.smithi110.stdout:(23/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 889 kB/s | 172 kB 00:00 2024-09-11T08:15:28.380 INFO:teuthology.orchestra.run.smithi110.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 407 kB/s | 20 kB 00:00 2024-09-11T08:15:28.398 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-11T08:15:28.405 INFO:teuthology.orchestra.run.smithi110.stdout:(25/34): python3-jaraco-collections-3.0.0-8.el9 172 kB/s | 23 kB 00:00 2024-09-11T08:15:28.431 INFO:teuthology.orchestra.run.smithi110.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-11T08:15:28.456 INFO:teuthology.orchestra.run.smithi110.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-11T08:15:28.481 INFO:teuthology.orchestra.run.smithi110.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-11T08:15:28.490 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-11T08:15:28.515 INFO:teuthology.orchestra.run.smithi110.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-11T08:15:28.539 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-11T08:15:28.540 INFO:teuthology.orchestra.run.smithi110.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-09-11T08:15:28.566 INFO:teuthology.orchestra.run.smithi110.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-11T08:15:28.575 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-11T08:15:28.582 INFO:teuthology.orchestra.run.smithi110.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-11T08:15:28.608 INFO:teuthology.orchestra.run.smithi110.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-11T08:15:28.614 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-11T08:15:28.641 INFO:teuthology.orchestra.run.smithi110.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-11T08:15:28.647 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:28.647 INFO:teuthology.orchestra.run.smithi110.stdout:Total 8.4 MB/s | 27 MB 00:03 2024-09-11T08:15:28.656 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-11T08:15:28.690 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-11T08:15:28.730 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-11T08:15:28.765 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-11T08:15:28.780 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:15:28.800 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-11T08:15:28.816 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:15:28.816 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:15:28.849 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-11T08:15:28.889 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T08:15:28.962 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-11T08:15:29.117 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-11T08:15:29.197 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-11T08:15:29.203 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:15:29.204 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:15:29.250 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-11T08:15:29.288 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-11T08:15:29.337 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-11T08:15:29.401 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-11T08:15:29.589 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-11T08:15:29.634 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-11T08:15:29.713 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-11T08:15:29.802 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-11T08:15:29.858 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-11T08:15:29.972 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-11T08:15:30.122 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 29/34 2024-09-11T08:15:30.152 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T08:15:30.178 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T08:15:30.179 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:30.179 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T08:15:30.179 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T08:15:30.179 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T08:15:30.179 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:30.437 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:15:30.503 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-11T08:15:30.544 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-11T08:15:30.576 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-11T08:15:30.669 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-11T08:15:30.738 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-11T08:15:30.784 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-11T08:15:30.829 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-11T08:15:30.858 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-11T08:15:30.893 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-11T08:15:30.926 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-11T08:15:30.992 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-11T08:15:31.026 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-11T08:15:31.061 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-11T08:15:31.110 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-11T08:15:31.142 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T08:15:31.215 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-11T08:15:31.359 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-11T08:15:31.441 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-11T08:15:31.495 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-11T08:15:31.502 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T08:15:31.530 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T08:15:31.530 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:31.530 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T08:15:31.530 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T08:15:31.530 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T08:15:31.530 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:31.533 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-11T08:15:31.582 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-11T08:15:31.638 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-11T08:15:31.822 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-11T08:15:31.879 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-11T08:15:31.903 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T08:15:31.929 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T08:15:31.929 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:31.929 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T08:15:31.929 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T08:15:31.929 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T08:15:31.929 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:31.949 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-11T08:15:32.037 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-11T08:15:32.083 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-11T08:15:32.129 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T08:15:32.153 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T08:15:32.153 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:32.153 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T08:15:32.153 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T08:15:32.153 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T08:15:32.153 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:32.201 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-11T08:15:32.217 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T08:15:32.349 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 29/34 2024-09-11T08:15:32.381 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T08:15:32.406 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 30/34 2024-09-11T08:15:32.406 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:32.406 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T08:15:32.406 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T08:15:32.406 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T08:15:32.406 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:33.757 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T08:15:33.787 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 31/34 2024-09-11T08:15:33.787 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:33.787 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T08:15:33.787 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T08:15:33.787 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T08:15:33.787 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 6/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-11T08:15:33.937 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-11T08:15:33.938 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-11T08:15:33.939 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-11T08:15:33.939 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-11T08:15:33.939 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-11T08:15:33.939 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-11T08:15:33.939 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-11T08:15:33.940 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-11T08:15:33.940 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-11T08:15:33.940 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-11T08:15:33.940 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-11T08:15:33.940 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-11T08:15:33.940 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-11T08:15:34.167 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T08:15:34.191 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 32/34 2024-09-11T08:15:34.191 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:34.191 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T08:15:34.192 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T08:15:34.192 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T08:15:34.192 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:34.392 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T08:15:34.416 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 33/34 2024-09-11T08:15:34.416 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:34.416 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T08:15:34.416 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T08:15:34.416 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T08:15:34.417 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T08:15:34.807 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-11T08:15:34.807 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:34.807 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:15:34.807 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:34.807 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T08:15:34.808 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T08:15:34.809 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T08:15:34.810 INFO:teuthology.orchestra.run.smithi139.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T08:15:34.810 INFO:teuthology.orchestra.run.smithi139.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T08:15:34.810 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T08:15:34.810 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T08:15:34.810 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T08:15:34.810 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:34.810 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:35.202 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-base 2024-09-11T08:15:35.773 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:50 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:35.865 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:15:35.906 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:35.907 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:15:35.908 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:35.972 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install cephadm 2024-09-11T08:15:36.243 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 34/34 2024-09-11T08:15:36.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/34 2024-09-11T08:15:36.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/34 2024-09-11T08:15:36.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 3/34 2024-09-11T08:15:36.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 4/34 2024-09-11T08:15:36.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 5/34 2024-09-11T08:15:36.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9. 6/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-11T08:15:36.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-11T08:15:36.245 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-11T08:15:36.245 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-11T08:15:36.245 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-11T08:15:36.245 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-11T08:15:36.245 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-11T08:15:36.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-11T08:15:36.526 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:36.658 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:36.659 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:36.659 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:15:36.659 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:36.659 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:15:36.659 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 83 k 2024-09-11T08:15:36.659 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:36.659 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:15:36.660 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:36.660 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:15:36.660 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:36.660 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 83 k 2024-09-11T08:15:36.660 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 375 k 2024-09-11T08:15:36.660 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:15:36.871 INFO:teuthology.orchestra.run.smithi139.stdout:cephadm-17.2.7-1623.gd0e6828a.el9.noarch.rpm 393 kB/s | 83 kB 00:00 2024-09-11T08:15:36.871 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:36.872 INFO:teuthology.orchestra.run.smithi139.stdout:Total 390 kB/s | 83 kB 00:00 2024-09-11T08:15:36.872 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:15:36.877 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:15:36.877 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:15:36.878 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:15:36.879 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:15:36.944 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:15:37.293 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-11T08:15:37.293 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:37.293 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T08:15:37.294 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T08:15:37.295 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:37.296 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:37.404 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:37.508 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:37.672 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base 2024-09-11T08:15:37.877 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:38.142 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:38.143 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:38.143 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:15:38.143 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:38.143 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:38.143 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:38.240 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:38.330 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:15:38.362 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T08:15:38.371 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:38.372 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:15:38.372 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:38.438 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install cephadm 2024-09-11T08:15:38.933 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:53 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:38.999 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:51 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:39.069 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:39.070 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:39.070 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-09-11T08:15:39.070 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:39.070 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:15:39.070 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 137 k 2024-09-11T08:15:39.070 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:39.070 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:15:39.071 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:39.071 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:15:39.071 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:39.071 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 137 k 2024-09-11T08:15:39.071 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 379 k 2024-09-11T08:15:39.071 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:15:39.131 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:39.132 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:39.132 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:15:39.132 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:39.132 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:15:39.132 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 83 k 2024-09-11T08:15:39.132 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:39.133 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:15:39.133 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:39.133 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:15:39.133 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:39.133 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 83 k 2024-09-11T08:15:39.133 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 375 k 2024-09-11T08:15:39.133 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:15:39.301 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-immutable-object-cache-17.2.7-1623.gd0e682 597 kB/s | 137 kB 00:00 2024-09-11T08:15:39.302 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:39.302 INFO:teuthology.orchestra.run.smithi139.stdout:Total 593 kB/s | 137 kB 00:00 2024-09-11T08:15:39.302 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:15:39.310 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:15:39.310 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:15:39.358 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm-17.2.7-1623.gd0e6828a.el9.noarch.rpm 367 kB/s | 83 kB 00:00 2024-09-11T08:15:39.359 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:39.359 INFO:teuthology.orchestra.run.smithi110.stdout:Total 365 kB/s | 83 kB 00:00 2024-09-11T08:15:39.359 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:15:39.364 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:15:39.364 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:15:39.366 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:15:39.367 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:15:39.367 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:15:39.368 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:15:39.445 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:15:39.511 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:15:39.577 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T08:15:39.601 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T08:15:39.601 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:39.601 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T08:15:39.602 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:39.916 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:40.003 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:40.391 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:40.508 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T08:15:40.508 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:40.508 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:15:40.508 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:40.508 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:40.509 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:40.631 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:15:40.632 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:40.632 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:15:40.632 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:40.632 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:40.632 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:40.720 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr 2024-09-11T08:15:40.821 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T08:15:41.300 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:41.390 INFO:teuthology.orchestra.run.smithi139.stdout:Package ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:15:41.397 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:54 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:41.431 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:41.433 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:15:41.433 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:41.497 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T08:15:41.531 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:41.531 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:41.531 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-11T08:15:41.531 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:41.531 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:15:41.531 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 137 k 2024-09-11T08:15:41.532 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:41.532 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:15:41.532 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:41.532 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:15:41.532 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:41.532 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 137 k 2024-09-11T08:15:41.533 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 379 k 2024-09-11T08:15:41.533 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:15:41.723 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-17.2.7-1623.gd0e682 720 kB/s | 137 kB 00:00 2024-09-11T08:15:41.724 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:41.724 INFO:teuthology.orchestra.run.smithi110.stdout:Total 714 kB/s | 137 kB 00:00 2024-09-11T08:15:41.725 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:15:41.732 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:15:41.732 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:15:41.789 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:15:41.789 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:15:41.952 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:15:42.049 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T08:15:42.057 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:00:57 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:42.074 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T08:15:42.074 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:15:42.074 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T08:15:42.074 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:42.191 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 3.4 M 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 24 k 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 15 k 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T08:15:42.192 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T08:15:42.193 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2024-09-11T08:15:42.193 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-11T08:15:42.193 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:42.193 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:15:42.193 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:42.193 INFO:teuthology.orchestra.run.smithi139.stdout:Install 7 Packages 2024-09-11T08:15:42.193 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:42.194 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 3.7 M 2024-09-11T08:15:42.194 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 80 M 2024-09-11T08:15:42.194 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:15:42.429 INFO:teuthology.orchestra.run.smithi139.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1623.gd0e 158 kB/s | 24 kB 00:00 2024-09-11T08:15:42.454 INFO:teuthology.orchestra.run.smithi139.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1623.gd0e6 81 kB/s | 15 kB 00:00 2024-09-11T08:15:42.496 INFO:teuthology.orchestra.run.smithi139.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-11T08:15:42.521 INFO:teuthology.orchestra.run.smithi139.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-11T08:15:42.546 INFO:teuthology.orchestra.run.smithi139.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-11T08:15:42.580 INFO:teuthology.orchestra.run.smithi139.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-11T08:15:42.722 INFO:teuthology.orchestra.run.smithi139.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1623.gd0e6828a 7.5 MB/s | 3.4 MB 00:00 2024-09-11T08:15:42.725 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:42.726 INFO:teuthology.orchestra.run.smithi139.stdout:Total 6.9 MB/s | 3.7 MB 00:00 2024-09-11T08:15:42.726 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:15:42.734 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:15:42.734 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:15:42.833 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:15:42.833 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:15:43.034 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a. 1/1 2024-09-11T08:15:43.034 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:43.034 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:15:43.034 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:15:43.034 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:43.034 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:43.132 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:15:43.198 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-11T08:15:43.232 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-11T08:15:43.254 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr 2024-09-11T08:15:43.287 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-11T08:15:43.327 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-11T08:15:43.356 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 5/7 2024-09-11T08:15:43.828 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:43.925 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-mgr-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:15:43.967 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:43.968 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:15:43.968 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:44.030 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T08:15:44.350 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 6/7 2024-09-11T08:15:44.401 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T08:15:44.600 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:57 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:44.733 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 3.4 M 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 24 k 2024-09-11T08:15:44.734 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 15 k 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout:Install 7 Packages 2024-09-11T08:15:44.735 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:44.736 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.7 M 2024-09-11T08:15:44.736 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 80 M 2024-09-11T08:15:44.736 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:15:45.070 INFO:teuthology.orchestra.run.smithi110.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1623.gd0e6 82 kB/s | 15 kB 00:00 2024-09-11T08:15:45.095 INFO:teuthology.orchestra.run.smithi110.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1623.gd0e 120 kB/s | 24 kB 00:00 2024-09-11T08:15:45.137 INFO:teuthology.orchestra.run.smithi110.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-11T08:15:45.162 INFO:teuthology.orchestra.run.smithi110.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-11T08:15:45.187 INFO:teuthology.orchestra.run.smithi110.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 445 kB/s | 41 kB 00:00 2024-09-11T08:15:45.221 INFO:teuthology.orchestra.run.smithi110.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-11T08:15:45.296 INFO:teuthology.orchestra.run.smithi110.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1623.gd0e6828a 8.3 MB/s | 3.4 MB 00:00 2024-09-11T08:15:45.300 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:45.300 INFO:teuthology.orchestra.run.smithi110.stdout:Total 6.5 MB/s | 3.7 MB 00:00 2024-09-11T08:15:45.300 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:15:45.308 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:15:45.309 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:15:45.395 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:15:45.395 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:15:45.520 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T08:15:45.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 1/7 2024-09-11T08:15:45.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 2/7 2024-09-11T08:15:45.520 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 3/7 2024-09-11T08:15:45.521 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-11T08:15:45.521 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-11T08:15:45.521 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-11T08:15:45.700 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:15:45.781 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-11T08:15:45.795 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-11T08:15:45.795 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:45.795 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:15:45.795 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:45.795 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:45.796 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:45.796 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-11T08:15:45.796 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-11T08:15:45.796 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T08:15:45.796 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T08:15:45.796 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:45.796 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:45.815 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-11T08:15:45.870 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-11T08:15:45.910 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-11T08:15:45.939 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 5/7 2024-09-11T08:15:45.992 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T08:15:46.552 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:46.686 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 7.4 M 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T08:15:46.687 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout:======================================================================================= 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout:Install 12 Packages 2024-09-11T08:15:46.688 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:46.689 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 38 M 2024-09-11T08:15:46.689 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 212 M 2024-09-11T08:15:46.689 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:15:46.930 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 6/7 2024-09-11T08:15:46.976 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T08:15:47.255 INFO:teuthology.orchestra.run.smithi139.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 17 MB/s | 7.4 MB 00:00 2024-09-11T08:15:47.338 INFO:teuthology.orchestra.run.smithi139.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 393 kB/s | 33 kB 00:00 2024-09-11T08:15:47.363 INFO:teuthology.orchestra.run.smithi139.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 346 kB/s | 191 kB 00:00 2024-09-11T08:15:47.439 INFO:teuthology.orchestra.run.smithi139.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 242 kB/s | 18 kB 00:00 2024-09-11T08:15:47.497 INFO:teuthology.orchestra.run.smithi139.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-11T08:15:47.556 INFO:teuthology.orchestra.run.smithi139.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-09-11T08:15:47.598 INFO:teuthology.orchestra.run.smithi139.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 5.7 MB/s | 245 kB 00:00 2024-09-11T08:15:47.673 INFO:teuthology.orchestra.run.smithi139.stdout:(8/12): libgfortran-11.5.0-2.el9.x86_64.rpm 929 kB/s | 801 kB 00:00 2024-09-11T08:15:47.740 INFO:teuthology.orchestra.run.smithi139.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-09-11T08:15:47.807 INFO:teuthology.orchestra.run.smithi139.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 16 MB/s | 4.9 MB 00:00 2024-09-11T08:15:47.915 INFO:teuthology.orchestra.run.smithi139.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.0 MB/s | 250 kB 00:00 2024-09-11T08:15:48.016 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 7/7 2024-09-11T08:15:48.016 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9. 1/7 2024-09-11T08:15:48.016 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarc 2/7 2024-09-11T08:15:48.017 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.n 3/7 2024-09-11T08:15:48.017 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-11T08:15:48.017 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-11T08:15:48.017 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-11T08:15:48.083 INFO:teuthology.orchestra.run.smithi139.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-11T08:15:48.083 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:48.083 INFO:teuthology.orchestra.run.smithi139.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-11T08:15:48.307 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-11T08:15:48.307 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:48.307 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:48.308 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:15:48.436 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:15:48.461 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:15:48.461 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:15:48.534 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T08:15:48.990 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:15:48.991 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:15:49.091 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:15:49.225 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 7.4 M 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T08:15:49.226 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-09-11T08:15:49.227 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:15:49.228 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 38 M 2024-09-11T08:15:49.228 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 212 M 2024-09-11T08:15:49.228 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:15:49.530 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-11T08:15:49.572 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-09-11T08:15:49.631 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.3 MB/s | 801 kB 00:00 2024-09-11T08:15:49.673 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-11T08:15:49.715 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 929 kB/s | 38 kB 00:00 2024-09-11T08:15:49.745 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:15:49.828 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T08:15:49.863 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T08:15:49.972 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T08:15:50.015 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T08:15:50.199 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T08:15:50.230 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T08:15:50.323 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 7.9 MB/s | 7.4 MB 00:00 2024-09-11T08:15:50.357 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T08:15:50.382 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-09-11T08:15:50.499 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-09-11T08:15:50.558 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-11T08:15:50.999 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T08:15:51.074 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.6 MB/s | 4.9 MB 00:01 2024-09-11T08:15:51.118 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T08:15:51.675 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.9 MB/s | 5.1 MB 00:01 2024-09-11T08:15:52.345 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T08:15:53.008 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-09-11T08:15:53.009 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:53.009 INFO:teuthology.orchestra.run.smithi110.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-11T08:15:53.362 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:15:53.377 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T08:15:53.388 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:15:53.388 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:15:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T08:15:53.936 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:15:53.936 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:15:54.741 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:15:54.836 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T08:15:54.873 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T08:15:54.974 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T08:15:55.016 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T08:15:55.200 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T08:15:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T08:15:55.329 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T08:15:55.963 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 1/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T08:15:56.000 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T08:15:56.001 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T08:15:56.088 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T08:15:56.430 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T08:15:56.430 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:56.430 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:15:56.430 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:56.431 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:15:56.930 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-rook 2024-09-11T08:15:57.325 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T08:15:57.499 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:12 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:15:57.633 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 47 k 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T08:15:57.634 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T08:15:57.635 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:57.636 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:15:57.636 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:15:57.636 INFO:teuthology.orchestra.run.smithi139.stdout:Install 13 Packages 2024-09-11T08:15:57.636 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:15:57.636 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 2.2 M 2024-09-11T08:15:57.637 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 26 M 2024-09-11T08:15:57.637 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:15:58.187 INFO:teuthology.orchestra.run.smithi139.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 232 kB/s | 19 kB 00:00 2024-09-11T08:15:58.212 INFO:teuthology.orchestra.run.smithi139.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 238 kB/s | 26 kB 00:00 2024-09-11T08:15:58.287 INFO:teuthology.orchestra.run.smithi139.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-11T08:15:58.312 INFO:teuthology.orchestra.run.smithi139.stdout:(4/13): ceph-mgr-rook-17.2.7-1623.gd0e6828a.el9 228 kB/s | 47 kB 00:00 2024-09-11T08:15:58.338 INFO:teuthology.orchestra.run.smithi139.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-11T08:15:58.357 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T08:15:58.363 INFO:teuthology.orchestra.run.smithi139.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-11T08:15:58.382 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T08:15:58.388 INFO:teuthology.orchestra.run.smithi139.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-11T08:15:58.439 INFO:teuthology.orchestra.run.smithi139.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-11T08:15:58.464 INFO:teuthology.orchestra.run.smithi139.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-11T08:15:58.489 INFO:teuthology.orchestra.run.smithi139.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-11T08:15:58.514 INFO:teuthology.orchestra.run.smithi139.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-11T08:15:58.539 INFO:teuthology.orchestra.run.smithi139.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-11T08:15:58.573 INFO:teuthology.orchestra.run.smithi139.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-11T08:15:58.576 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:15:58.576 INFO:teuthology.orchestra.run.smithi139.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-11T08:15:58.642 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:15:58.656 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:15:58.657 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:15:58.744 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:15:58.746 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:15:59.257 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:15:59.418 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T08:15:59.475 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T08:15:59.538 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T08:15:59.575 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T08:15:59.616 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T08:15:59.718 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T08:15:59.820 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T08:15:59.876 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T08:16:00.585 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T08:16:00.640 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T08:16:00.674 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T08:16:00.719 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T08:16:00.743 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T08:16:00.966 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 12/12 2024-09-11T08:16:00.968 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e68 1/12 2024-09-11T08:16:00.968 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T08:16:00.968 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T08:16:00.968 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T08:16:00.968 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T08:16:00.968 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T08:16:00.968 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T08:16:00.969 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T08:16:00.969 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T08:16:00.969 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T08:16:00.969 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T08:16:01.364 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T08:16:01.364 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:01.364 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:01.364 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:16:01.364 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T08:16:01.364 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T08:16:01.364 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:01.365 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:01.486 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T08:16:01.486 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 1/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T08:16:01.487 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T08:16:01.488 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T08:16:01.488 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T08:16:01.722 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-rook 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T08:16:01.837 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:01.838 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:02.082 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T08:16:02.293 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:15 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:02.433 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 47 k 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T08:16:02.434 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout:Install 13 Packages 2024-09-11T08:16:02.435 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:02.436 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.2 M 2024-09-11T08:16:02.436 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-09-11T08:16:02.436 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:02.657 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:17 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:02.793 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:02.794 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 124 k 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:02.795 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:02.796 INFO:teuthology.orchestra.run.smithi139.stdout:Install 6 Packages 2024-09-11T08:16:02.796 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:02.796 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 7.0 M 2024-09-11T08:16:02.796 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 33 M 2024-09-11T08:16:02.796 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:02.816 INFO:teuthology.orchestra.run.smithi110.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 313 kB/s | 26 kB 00:00 2024-09-11T08:16:02.841 INFO:teuthology.orchestra.run.smithi110.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 179 kB/s | 19 kB 00:00 2024-09-11T08:16:02.900 INFO:teuthology.orchestra.run.smithi110.stdout:(3/13): ceph-mgr-rook-17.2.7-1623.gd0e6828a.el9 286 kB/s | 47 kB 00:00 2024-09-11T08:16:02.950 INFO:teuthology.orchestra.run.smithi110.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-11T08:16:02.975 INFO:teuthology.orchestra.run.smithi110.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-11T08:16:03.009 INFO:teuthology.orchestra.run.smithi110.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-09-11T08:16:03.043 INFO:teuthology.orchestra.run.smithi110.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-11T08:16:03.068 INFO:teuthology.orchestra.run.smithi110.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-11T08:16:03.093 INFO:teuthology.orchestra.run.smithi110.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-11T08:16:03.118 INFO:teuthology.orchestra.run.smithi110.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-11T08:16:03.144 INFO:teuthology.orchestra.run.smithi110.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-11T08:16:03.186 INFO:teuthology.orchestra.run.smithi110.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-09-11T08:16:03.236 INFO:teuthology.orchestra.run.smithi110.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 766 kB/s | 90 kB 00:00 2024-09-11T08:16:03.239 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:03.239 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-11T08:16:03.266 INFO:teuthology.orchestra.run.smithi139.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-11T08:16:03.303 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:03.318 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:03.318 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:03.325 INFO:teuthology.orchestra.run.smithi139.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-11T08:16:03.350 INFO:teuthology.orchestra.run.smithi139.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1623.gd0e6828a.e 538 kB/s | 124 kB 00:00 2024-09-11T08:16:03.375 INFO:teuthology.orchestra.run.smithi139.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-11T08:16:03.406 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:03.407 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:03.417 INFO:teuthology.orchestra.run.smithi139.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-11T08:16:03.843 INFO:teuthology.orchestra.run.smithi139.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.3 MB/s | 6.0 MB 00:00 2024-09-11T08:16:03.845 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:03.845 INFO:teuthology.orchestra.run.smithi139.stdout:Total 6.7 MB/s | 7.0 MB 00:01 2024-09-11T08:16:03.910 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:03.916 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:03.928 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:03.928 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:04.026 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:04.026 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:04.082 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T08:16:04.137 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T08:16:04.193 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T08:16:04.230 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T08:16:04.271 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T08:16:04.327 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:04.373 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T08:16:04.474 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T08:16:04.493 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T08:16:04.523 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T08:16:04.553 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T08:16:05.082 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T08:16:05.173 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T08:16:05.235 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T08:16:05.236 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T08:16:05.265 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T08:16:05.279 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T08:16:05.313 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T08:16:05.358 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T08:16:05.383 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T08:16:06.097 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T08:16:06.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/6 2024-09-11T08:16:06.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T08:16:06.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T08:16:06.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T08:16:06.098 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T08:16:06.206 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 13/13 2024-09-11T08:16:06.206 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 1/13 2024-09-11T08:16:06.206 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T08:16:06.206 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T08:16:06.206 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T08:16:06.206 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T08:16:06.206 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T08:16:06.207 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T08:16:06.207 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T08:16:06.207 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T08:16:06.207 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T08:16:06.207 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T08:16:06.208 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T08:16:06.382 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T08:16:06.382 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:06.382 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:06.382 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:16:06.382 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T08:16:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T08:16:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T08:16:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T08:16:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T08:16:06.383 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:06.383 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T08:16:06.532 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:06.533 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:06.615 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-fuse 2024-09-11T08:16:06.767 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T08:16:07.172 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:22 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:07.303 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:07.303 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 813 k 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout:Install 2 Packages 2024-09-11T08:16:07.304 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:07.305 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 893 k 2024-09-11T08:16:07.305 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 2.6 M 2024-09-11T08:16:07.305 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:07.332 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:20 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:07.466 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:07.466 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:07.466 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:16:07.466 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 124 k 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-09-11T08:16:07.467 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:07.468 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 7.0 M 2024-09-11T08:16:07.468 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 33 M 2024-09-11T08:16:07.468 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:07.711 INFO:teuthology.orchestra.run.smithi139.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 618 kB/s | 80 kB 00:00 2024-09-11T08:16:07.878 INFO:teuthology.orchestra.run.smithi139.stdout:(2/2): ceph-fuse-17.2.7-1623.gd0e6828a.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-11T08:16:07.878 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:07.878 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-09-11T08:16:07.890 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:07.912 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:07.912 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:07.994 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:07.995 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:08.056 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-11T08:16:08.114 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-11T08:16:08.140 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1623.gd0e6828a.e 569 kB/s | 124 kB 00:00 2024-09-11T08:16:08.165 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-11T08:16:08.181 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:08.207 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-11T08:16:08.282 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-11T08:16:08.284 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:08.285 INFO:teuthology.orchestra.run.smithi110.stdout:Total 8.6 MB/s | 7.0 MB 00:00 2024-09-11T08:16:08.323 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T08:16:08.356 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:08.368 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:08.368 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:08.401 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T08:16:08.468 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:08.468 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:08.772 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:08.954 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T08:16:09.008 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T08:16:09.071 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T08:16:09.071 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/2 2024-09-11T08:16:09.361 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T08:16:09.361 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:09.361 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:09.361 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T08:16:09.361 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:09.362 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:09.524 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T08:16:09.611 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T08:16:09.631 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-volume 2024-09-11T08:16:09.667 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T08:16:09.696 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T08:16:10.202 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:25 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:10.338 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:10.338 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:10.338 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2024-09-11T08:16:10.338 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 270 k 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 270 k 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 1.3 M 2024-09-11T08:16:10.339 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:10.494 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 6/6 2024-09-11T08:16:10.494 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 1/6 2024-09-11T08:16:10.494 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T08:16:10.494 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T08:16:10.494 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T08:16:10.494 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T08:16:10.565 INFO:teuthology.orchestra.run.smithi139.stdout:ceph-volume-17.2.7-1623.gd0e6828a.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-11T08:16:10.566 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:10.566 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-11T08:16:10.566 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:10.575 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:10.576 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:10.605 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:10.606 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:10.762 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T08:16:10.762 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:10.762 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:10.762 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:16:10.762 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T08:16:10.763 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T08:16:10.763 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T08:16:10.763 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T08:16:10.763 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T08:16:10.763 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:10.763 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:10.816 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:10.884 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:16:10.902 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:16:10.902 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:16:10.902 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T08:16:10.902 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:11.008 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse 2024-09-11T08:16:11.576 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:11.711 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:11.711 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:11.711 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 813 k 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-09-11T08:16:11.712 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:11.713 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 893 k 2024-09-11T08:16:11.713 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.6 M 2024-09-11T08:16:11.713 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:11.950 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:16:11.950 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:11.951 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:11.951 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:16:11.951 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:11.951 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:12.009 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 457 kB/s | 80 kB 00:00 2024-09-11T08:16:12.143 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-fuse-17.2.7-1623.gd0e6828a.el9.x86_ 2.6 MB/s | 813 kB 00:00 2024-09-11T08:16:12.143 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:12.143 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.0 MB/s | 893 kB 00:00 2024-09-11T08:16:12.155 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:12.177 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:12.177 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:12.224 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados-devel 2024-09-11T08:16:12.261 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:12.261 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:12.454 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:12.599 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T08:16:12.667 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T08:16:12.813 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:12.949 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:12.951 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:12.951 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-09-11T08:16:12.951 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:12.951 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:12.952 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 126 k 2024-09-11T08:16:12.952 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:12.952 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:12.952 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:12.952 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:16:12.952 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:12.953 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 126 k 2024-09-11T08:16:12.953 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 448 k 2024-09-11T08:16:12.953 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:13.162 INFO:teuthology.orchestra.run.smithi139.stdout:librados-devel-17.2.7-1623.gd0e6828a.el9.x86_64 600 kB/s | 126 kB 00:00 2024-09-11T08:16:13.162 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:13.163 INFO:teuthology.orchestra.run.smithi139.stdout:Total 595 kB/s | 126 kB 00:00 2024-09-11T08:16:13.163 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:13.168 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:13.168 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:13.231 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:13.231 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:13.344 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2/2 2024-09-11T08:16:13.344 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/2 2024-09-11T08:16:13.377 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:13.493 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T08:16:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:13.588 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T08:16:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:13.588 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:13.860 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-volume 2024-09-11T08:16:13.927 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:14.186 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:14.186 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:14.186 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:14.186 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:14.186 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:14.186 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:14.443 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs2 2024-09-11T08:16:14.448 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:14.586 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:17.2.7-1623.gd0e6828a.el9 ceph-noarch 270 k 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:14.587 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:16:14.588 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:14.588 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 270 k 2024-09-11T08:16:14.588 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.3 M 2024-09-11T08:16:14.588 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:14.807 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-volume-17.2.7-1623.gd0e6828a.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-11T08:16:14.807 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:14.808 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-11T08:16:14.808 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:14.816 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:14.816 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:14.844 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:14.844 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:15.015 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:30 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:15.066 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:15.106 INFO:teuthology.orchestra.run.smithi139.stdout:Package libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:15.151 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:15.151 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:16:15.152 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:16:15.152 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:15.167 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:16:15.167 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:16:15.167 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T08:16:15.167 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:15.217 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install libcephfs-devel 2024-09-11T08:16:15.780 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:30 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:15.914 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:15.914 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 25 k 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 110 k 2024-09-11T08:16:15.915 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:16.057 INFO:teuthology.orchestra.run.smithi139.stdout:libcephfs-devel-17.2.7-1623.gd0e6828a.el9.x86_6 177 kB/s | 25 kB 00:00 2024-09-11T08:16:16.057 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:16.057 INFO:teuthology.orchestra.run.smithi139.stdout:Total 175 kB/s | 25 kB 00:00 2024-09-11T08:16:16.058 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:16.061 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:16.061 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:16.062 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:16.062 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:16.147 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 1/1 2024-09-11T08:16:16.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:16.147 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:16.147 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:17.2.7-1623.gd0e6828a.el9.noarch 2024-09-11T08:16:16.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:16.147 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:16.148 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:16.262 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:16.448 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel 2024-09-11T08:16:16.626 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:16.856 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:16.856 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:16.856 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:16.856 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:16.856 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:16.856 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:17.027 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librados2 2024-09-11T08:16:17.032 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:30 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:17.169 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:17.170 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:17.170 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-11T08:16:17.170 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:17.170 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:17.170 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 126 k 2024-09-11T08:16:17.170 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:17.171 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:17.171 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:17.171 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:16:17.171 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:17.171 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 126 k 2024-09-11T08:16:17.171 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 448 k 2024-09-11T08:16:17.171 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:17.380 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-17.2.7-1623.gd0e6828a.el9.x86_64 606 kB/s | 126 kB 00:00 2024-09-11T08:16:17.380 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:17.381 INFO:teuthology.orchestra.run.smithi110.stdout:Total 601 kB/s | 126 kB 00:00 2024-09-11T08:16:17.381 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:17.386 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:17.386 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:17.449 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:17.449 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:17.586 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:17.598 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:17.675 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:17.702 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:17.719 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:17.720 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:16:17.720 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:17.780 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install librbd1 2024-09-11T08:16:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:18.335 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:18.387 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:18.388 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:18.388 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:18.388 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:18.388 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:18.388 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:18.424 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:18.468 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:18.469 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:16:18.469 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:18.528 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rados 2024-09-11T08:16:18.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2 2024-09-11T08:16:19.087 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:19.177 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:19.220 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:19.221 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:16:19.221 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:19.237 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:19.289 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rgw 2024-09-11T08:16:19.327 INFO:teuthology.orchestra.run.smithi110.stdout:Package libcephfs2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:19.369 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:19.370 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:16:19.370 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:19.432 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs-devel 2024-09-11T08:16:19.850 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:19.940 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:19.983 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:19.984 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:16:19.984 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:19.995 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:20.056 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-cephfs 2024-09-11T08:16:20.129 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 25 k 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:20.130 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:20.131 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:16:20.131 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:20.131 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 25 k 2024-09-11T08:16:20.131 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 110 k 2024-09-11T08:16:20.131 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:20.284 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs-devel-17.2.7-1623.gd0e6828a.el9.x86_6 163 kB/s | 25 kB 00:00 2024-09-11T08:16:20.284 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:20.285 INFO:teuthology.orchestra.run.smithi110.stdout:Total 162 kB/s | 25 kB 00:00 2024-09-11T08:16:20.285 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:20.288 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:20.288 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:20.290 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:20.290 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:20.375 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:20.489 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:20.618 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:20.708 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:20.751 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:20.751 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:16:20.752 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:20.810 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install python3-rbd 2024-09-11T08:16:20.863 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:21.088 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:21.088 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:21.257 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2 2024-09-11T08:16:21.369 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:21.460 INFO:teuthology.orchestra.run.smithi139.stdout:Package python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:21.503 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:21.504 INFO:teuthology.orchestra.run.smithi139.stdout:Nothing to do. 2024-09-11T08:16:21.504 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:21.569 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-fuse 2024-09-11T08:16:21.817 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:21.906 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:21.950 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:21.951 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:16:21.951 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:22.015 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1 2024-09-11T08:16:22.134 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:22.268 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:22.269 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:22.269 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-09-11T08:16:22.269 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 82 k 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 82 k 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 222 k 2024-09-11T08:16:22.270 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:22.468 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-fuse-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 413 kB/s | 82 kB 00:00 2024-09-11T08:16:22.469 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:22.469 INFO:teuthology.orchestra.run.smithi139.stdout:Total 411 kB/s | 82 kB 00:00 2024-09-11T08:16:22.469 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:22.476 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:22.476 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:22.538 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:22.538 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:22.566 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:22.656 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:22.680 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:22.698 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:22.699 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:16:22.699 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:22.761 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados 2024-09-11T08:16:22.781 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:23.174 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:23.317 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:23.406 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rados-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:23.415 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:23.415 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:23.415 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:23.415 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:23.415 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:23.416 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:23.449 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:23.450 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:16:23.450 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:23.515 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw 2024-09-11T08:16:23.625 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-mirror 2024-09-11T08:16:24.074 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:24.164 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rgw-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:24.200 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:24.207 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:24.208 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:16:24.208 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:24.270 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs 2024-09-11T08:16:24.335 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repo Size 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:16:24.336 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:24.337 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 2.9 M 2024-09-11T08:16:24.337 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 12 M 2024-09-11T08:16:24.337 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:24.646 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-mirror-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 9.5 MB/s | 2.9 MB 00:00 2024-09-11T08:16:24.647 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:24.647 INFO:teuthology.orchestra.run.smithi139.stdout:Total 9.4 MB/s | 2.9 MB 00:00 2024-09-11T08:16:24.648 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:24.656 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:24.656 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:24.735 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:24.736 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:24.829 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:24.918 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-cephfs-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:24.961 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:24.962 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:16:24.962 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:25.020 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd 2024-09-11T08:16:25.145 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:25.206 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:25.230 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:25.230 INFO:teuthology.orchestra.run.smithi139.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:16:25.230 INFO:teuthology.orchestra.run.smithi139.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T08:16:25.230 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:16:25.230 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:16:25.230 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:25.579 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:38 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:25.670 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 is already installed. 2024-09-11T08:16:25.713 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:25.714 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-11T08:16:25.714 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:25.788 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse 2024-09-11T08:16:26.273 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:26.273 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:26.273 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:26.273 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:26.273 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:26.273 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:26.340 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:26.471 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 82 k 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:26.472 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:26.473 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:16:26.473 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:26.473 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 82 k 2024-09-11T08:16:26.473 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 222 k 2024-09-11T08:16:26.473 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:26.505 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install rbd-nbd 2024-09-11T08:16:26.679 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 397 kB/s | 82 kB 00:00 2024-09-11T08:16:26.680 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:26.680 INFO:teuthology.orchestra.run.smithi110.stdout:Total 394 kB/s | 82 kB 00:00 2024-09-11T08:16:26.680 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:26.686 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:26.687 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:26.749 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:26.750 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:26.889 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:27.009 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:27.079 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Sep 2024 08:14:45 AM UTC. 2024-09-11T08:16:27.211 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 165 k 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2024-09-11T08:16:27.212 INFO:teuthology.orchestra.run.smithi139.stdout:Install 1 Package 2024-09-11T08:16:27.213 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:27.213 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 165 k 2024-09-11T08:16:27.213 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 477 k 2024-09-11T08:16:27.213 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2024-09-11T08:16:27.384 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:27.424 INFO:teuthology.orchestra.run.smithi139.stdout:rbd-nbd-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 779 kB/s | 165 kB 00:00 2024-09-11T08:16:27.425 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:27.425 INFO:teuthology.orchestra.run.smithi139.stdout:Total 774 kB/s | 165 kB 00:00 2024-09-11T08:16:27.426 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2024-09-11T08:16:27.433 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2024-09-11T08:16:27.433 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2024-09-11T08:16:27.497 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2024-09-11T08:16:27.497 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2024-09-11T08:16:27.603 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:27.603 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:27.603 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:27.603 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:27.603 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:27.603 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:27.644 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2024-09-11T08:16:27.747 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:27.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror 2024-09-11T08:16:28.165 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:28.413 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:28.440 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:28.440 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:28.441 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2024-09-11T08:16:28.441 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:28.441 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:16:28.441 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2024-09-11T08:16:28.549 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:28.549 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 2.9 M 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.9 M 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 12 M 2024-09-11T08:16:28.550 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:28.662 DEBUG:teuthology.parallel:result is None 2024-09-11T08:16:28.891 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 8.6 MB/s | 2.9 MB 00:00 2024-09-11T08:16:28.892 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:28.892 INFO:teuthology.orchestra.run.smithi110.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-09-11T08:16:28.892 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:28.901 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:28.901 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:28.983 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:28.983 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:29.389 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:29.450 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:29.475 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:29.475 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T08:16:29.475 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T08:16:29.476 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:16:29.476 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:16:29.476 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:30.506 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:30.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:30.507 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:30.507 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:30.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:30.507 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:30.719 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd 2024-09-11T08:16:31.298 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Sep 2024 08:14:47 AM UTC. 2024-09-11T08:16:31.432 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-11T08:16:31.433 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:31.433 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-11T08:16:31.433 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:31.433 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-11T08:16:31.433 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:17.2.7-1623.gd0e6828a.el9 ceph 165 k 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 165 k 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 477 k 2024-09-11T08:16:31.434 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-11T08:16:31.637 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-17.2.7-1623.gd0e6828a.el9.x86_64.rpm 814 kB/s | 165 kB 00:00 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi110.stdout:Total 808 kB/s | 165 kB 00:00 2024-09-11T08:16:31.638 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-11T08:16:31.645 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-11T08:16:31.709 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-11T08:16:31.709 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-11T08:16:31.873 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-11T08:16:31.992 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:32.638 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 1/1 2024-09-11T08:16:32.639 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:32.639 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-11T08:16:32.639 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:17.2.7-1623.gd0e6828a.el9.x86_64 2024-09-11T08:16:32.639 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:16:32.639 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-11T08:16:32.896 DEBUG:teuthology.parallel:result is None 2024-09-11T08:16:32.897 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:16:33.059 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T08:16:33.080 INFO:teuthology.orchestra.run.smithi110.stdout:17.2.7-1623.gd0e6828a.el9 2024-09-11T08:16:33.081 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1623.gd0e6828a.el9 2024-09-11T08:16:33.081 INFO:teuthology.task.install:The correct ceph version 17.2.7-1623.gd0e6828a is installed. 2024-09-11T08:16:33.083 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:16:33.232 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T08:16:33.257 INFO:teuthology.orchestra.run.smithi139.stdout:17.2.7-1623.gd0e6828a.el9 2024-09-11T08:16:33.257 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1623.gd0e6828a.el9 2024-09-11T08:16:33.257 INFO:teuthology.task.install:The correct ceph version 17.2.7-1623.gd0e6828a is installed. 2024-09-11T08:16:33.259 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T08:16:33.259 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:16:33.259 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T08:16:33.286 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:16:33.286 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T08:16:33.327 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T08:16:33.327 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:16:33.327 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T08:16:33.351 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T08:16:33.415 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:16:33.416 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T08:16:33.446 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T08:16:33.513 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T08:16:33.513 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:16:33.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T08:16:33.538 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T08:16:33.604 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:16:33.605 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T08:16:33.630 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T08:16:33.695 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T08:16:33.695 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:16:33.695 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T08:16:33.720 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T08:16:33.784 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:16:33.785 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T08:16:33.809 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T08:16:33.875 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-11T08:16:33.974 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'd0e6828a2016d48cf25ad84064e50742bb1c39b9', 'cephadm_mode': 'root'} 2024-09-11T08:16:33.975 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:16:33.975 INFO:tasks.cephadm:Cluster fsid is 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:16:33.975 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-11T08:16:33.975 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.110', 'mon.c': '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]', 'mon.b': '172.21.15.139'} 2024-09-11T08:16:33.976 INFO:tasks.cephadm:First mon is mon.a on smithi110 2024-09-11T08:16:33.976 INFO:tasks.cephadm:First mgr is y 2024-09-11T08:16:33.976 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-11T08:16:33.976 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname $(hostname -s) 2024-09-11T08:16:34.001 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2024-09-11T08:16:34.026 INFO:tasks.cephadm:Downloading cephadm (repo https://git.ceph.com/ceph-ci.git ref d0e6828a2016d48cf25ad84064e50742bb1c39b9)... 2024-09-11T08:16:34.027 DEBUG:teuthology.orchestra.run.smithi110:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show d0e6828a2016d48cf25ad84064e50742bb1c39b9:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-11T08:16:34.057 INFO:teuthology.orchestra.run.smithi110.stderr:Cloning into 'testrepo'... 2024-09-11T08:17:31.876 INFO:teuthology.orchestra.run.smithi110.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 11 08:17 /home/ubuntu/cephtest/cephadm 2024-09-11T08:17:31.877 DEBUG:teuthology.orchestra.run.smithi139:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show d0e6828a2016d48cf25ad84064e50742bb1c39b9:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-11T08:17:31.896 INFO:teuthology.orchestra.run.smithi139.stderr:Cloning into 'testrepo'... 2024-09-11T08:18:28.440 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 11 08:18 /home/ubuntu/cephtest/cephadm 2024-09-11T08:18:28.441 DEBUG:teuthology.orchestra.run.smithi110:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-11T08:18:28.462 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-11T08:18:28.508 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 on all hosts... 2024-09-11T08:18:28.508 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 pull 2024-09-11T08:18:28.511 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 pull 2024-09-11T08:18:30.096 INFO:teuthology.orchestra.run.smithi110.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:18:31.807 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:19:01.558 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2024-09-11T08:19:01.558 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_version": "ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable)", 2024-09-11T08:19:01.558 INFO:teuthology.orchestra.run.smithi139.stdout: "image_id": "4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b", 2024-09-11T08:19:01.558 INFO:teuthology.orchestra.run.smithi139.stdout: "repo_digests": [ 2024-09-11T08:19:01.559 INFO:teuthology.orchestra.run.smithi139.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40" 2024-09-11T08:19:01.559 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2024-09-11T08:19:01.559 INFO:teuthology.orchestra.run.smithi139.stdout:} 2024-09-11T08:19:02.074 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-11T08:19:02.074 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable)", 2024-09-11T08:19:02.074 INFO:teuthology.orchestra.run.smithi110.stdout: "image_id": "4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b", 2024-09-11T08:19:02.074 INFO:teuthology.orchestra.run.smithi110.stdout: "repo_digests": [ 2024-09-11T08:19:02.075 INFO:teuthology.orchestra.run.smithi110.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40" 2024-09-11T08:19:02.075 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-11T08:19:02.075 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-11T08:19:02.093 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph 2024-09-11T08:19:02.139 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2024-09-11T08:19:02.172 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /etc/ceph 2024-09-11T08:19:02.196 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2024-09-11T08:19:02.241 INFO:tasks.cephadm:Writing seed config... 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-11T08:19:02.243 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-11T08:19:02.244 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-11T08:19:02.244 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-11T08:19:02.244 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-11T08:19:02.244 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-11T08:19:02.244 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:19:02.244 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-11T08:19:02.260 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2850e098-7016-11ef-bcea-c7b262605968 mon election default strategy = 3 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-11T08:19:02.260 DEBUG:teuthology.orchestra.run.smithi110:mon.a> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a.service 2024-09-11T08:19:02.303 DEBUG:teuthology.orchestra.run.smithi110:mgr.y> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service 2024-09-11T08:19:02.346 INFO:tasks.cephadm:Bootstrapping... 2024-09-11T08:19:02.347 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 -v bootstrap --fsid 2850e098-7016-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.110 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:19:02.545 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-11T08:19:02.545 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9', '-v', 'bootstrap', '--fsid', '2850e098-7016-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.110', '--skip-admin-label'] 2024-09-11T08:19:02.581 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stdout 5.2.2 2024-09-11T08:19:02.581 INFO:teuthology.orchestra.run.smithi110.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-11T08:19:02.581 INFO:teuthology.orchestra.run.smithi110.stdout:Verifying podman|docker is present... 2024-09-11T08:19:02.602 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stdout 5.2.2 2024-09-11T08:19:02.602 INFO:teuthology.orchestra.run.smithi110.stdout:Verifying lvm2 is present... 2024-09-11T08:19:02.603 INFO:teuthology.orchestra.run.smithi110.stdout:Verifying time synchronization is in place... 2024-09-11T08:19:02.611 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-11T08:19:02.611 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-11T08:19:02.617 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-11T08:19:02.617 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout inactive 2024-09-11T08:19:02.624 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout enabled 2024-09-11T08:19:02.631 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout active 2024-09-11T08:19:02.631 INFO:teuthology.orchestra.run.smithi110.stdout:Unit chronyd.service is enabled and running 2024-09-11T08:19:02.631 INFO:teuthology.orchestra.run.smithi110.stdout:Repeating the final host check... 2024-09-11T08:19:02.660 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stdout 5.2.2 2024-09-11T08:19:02.660 INFO:teuthology.orchestra.run.smithi110.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-11T08:19:02.660 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl is present 2024-09-11T08:19:02.661 INFO:teuthology.orchestra.run.smithi110.stdout:lvcreate is present 2024-09-11T08:19:02.667 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-11T08:19:02.667 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-11T08:19:02.673 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-11T08:19:02.674 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout inactive 2024-09-11T08:19:02.680 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout enabled 2024-09-11T08:19:02.688 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout active 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Unit chronyd.service is enabled and running 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Host looks OK 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Cluster fsid: 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Acquiring lock 140156355404704 on /run/cephadm/2850e098-7016-11ef-bcea-c7b262605968.lock 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Lock 140156355404704 acquired on /run/cephadm/2850e098-7016-11ef-bcea-c7b262605968.lock 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Verifying IP 172.21.15.110 port 3300 ... 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Verifying IP 172.21.15.110 port 6789 ... 2024-09-11T08:19:02.689 INFO:teuthology.orchestra.run.smithi110.stdout:Base mon IP(s) is [172.21.15.110:3300, 172.21.15.110:6789], mon addrv is [v2:172.21.15.110:3300,v1:172.21.15.110:6789] 2024-09-11T08:19:02.692 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.110 metric 100 2024-09-11T08:19:02.692 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.110 metric 100 2024-09-11T08:19:02.695 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-11T08:19:02.695 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-11T08:19:02.695 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-11T08:19:02.698 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-11T08:19:02.698 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-11T08:19:02.698 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-11T08:19:02.698 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-11T08:19:02.698 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cc35/64 scope link noprefixroute 2024-09-11T08:19:02.699 INFO:teuthology.orchestra.run.smithi110.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-11T08:19:02.699 INFO:teuthology.orchestra.run.smithi110.stdout:Mon IP `172.21.15.110` is in CIDR network `172.21.0.0/20` 2024-09-11T08:19:02.699 INFO:teuthology.orchestra.run.smithi110.stdout:Mon IP `172.21.15.110` is in CIDR network `172.21.0.0/20` 2024-09-11T08:19:02.699 INFO:teuthology.orchestra.run.smithi110.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-11T08:19:02.699 INFO:teuthology.orchestra.run.smithi110.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-11T08:19:02.700 INFO:teuthology.orchestra.run.smithi110.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:19:03.257 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stdout 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b 2024-09-11T08:19:03.258 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9... 2024-09-11T08:19:03.258 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stderr Getting image source signatures 2024-09-11T08:19:03.258 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stderr Copying blob sha256:0d2f264d63cec2f6fb0f85eed7b2c8c5b4db3ab9edd9d71a74c04647294a553f 2024-09-11T08:19:03.258 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-11T08:19:03.258 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stderr Copying config sha256:4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b 2024-09-11T08:19:03.258 INFO:teuthology.orchestra.run.smithi110.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-11T08:19:04.120 INFO:teuthology.orchestra.run.smithi110.stdout:ceph: stdout ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable) 2024-09-11T08:19:04.120 INFO:teuthology.orchestra.run.smithi110.stdout:Ceph version: ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable) 2024-09-11T08:19:04.120 INFO:teuthology.orchestra.run.smithi110.stdout:Extracting ceph user uid/gid from container image... 2024-09-11T08:19:05.102 INFO:teuthology.orchestra.run.smithi110.stdout:stat: stdout 167 167 2024-09-11T08:19:05.102 INFO:teuthology.orchestra.run.smithi110.stdout:Creating initial keys... 2024-09-11T08:19:06.049 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph-authtool: stdout AQB5UuFmZ5dIFxAAgnCoN6eVUfb8K4G557UW2w== 2024-09-11T08:19:07.055 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph-authtool: stdout AQB6UuFmcqyeFRAAuPu1HwqdSen2V42tDmoZSw== 2024-09-11T08:19:08.112 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph-authtool: stdout AQB7UuFmx1v8GRAATpUkamblIVyfy7J+D37c5A== 2024-09-11T08:19:08.113 INFO:teuthology.orchestra.run.smithi110.stdout:Creating initial monmap... 2024-09-11T08:19:09.061 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-11T08:19:09.061 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-11T08:19:09.061 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:09.061 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-11T08:19:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool for a [v2:172.21.15.110:3300,v1:172.21.15.110:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-11T08:19:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = octopus 2024-09-11T08:19:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/monmaptool: set fsid to 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-11T08:19:09.062 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:19:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:Creating mon... 2024-09-11T08:19:10.007 INFO:teuthology.orchestra.run.smithi110.stdout:create mon.a on 2024-09-11T08:19:10.215 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-11T08:19:10.372 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-11T08:19:10.518 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2850e098-7016-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-2850e098-7016-11ef-bcea-c7b262605968.target. 2024-09-11T08:19:10.518 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2850e098-7016-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-2850e098-7016-11ef-bcea-c7b262605968.target. 2024-09-11T08:19:10.713 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a 2024-09-11T08:19:10.713 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-11T08:19:10.894 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2850e098-7016-11ef-bcea-c7b262605968.target.wants/ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a.service → /etc/systemd/system/ceph-2850e098-7016-11ef-bcea-c7b262605968@.service. 2024-09-11T08:19:11.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:11 smithi110 podman[28614]: 2024-09-11 08:19:11.027126769 +0000 UTC m=+0.015339446 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:19:11.416 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:19:11.416 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout disabled 2024-09-11T08:19:11.424 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:19:11.424 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout inactive 2024-09-11T08:19:11.424 INFO:teuthology.orchestra.run.smithi110.stdout:firewalld.service is not enabled 2024-09-11T08:19:11.424 INFO:teuthology.orchestra.run.smithi110.stdout:Not possible to enable service . firewalld.service is not available 2024-09-11T08:19:11.425 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for mon to start... 2024-09-11T08:19:11.425 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for mon... 2024-09-11T08:19:11.505 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:11 smithi110 podman[28614]: 2024-09-11 08:19:11.343524782 +0000 UTC m=+0.331737464 container init 58cad7d83df78b52e96f429f3f89d21892c55b5923f617f1e744014109a4f32d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:19:11.505 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:11 smithi110 podman[28614]: 2024-09-11 08:19:11.346632541 +0000 UTC m=+0.334845219 container start 58cad7d83df78b52e96f429f3f89d21892c55b5923f617f1e744014109a4f32d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-11T08:19:11.505 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:11 smithi110 ceph-mon[28633]: mkfs 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:11.505 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:11 smithi110 ceph-mon[28633]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-11T08:19:11.505 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:11 smithi110 bash[28614]: 58cad7d83df78b52e96f429f3f89d21892c55b5923f617f1e744014109a4f32d 2024-09-11T08:19:11.505 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:11 smithi110 systemd[1]: Started Ceph mon.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:19:12.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:12 smithi110 ceph-mon[28633]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-11T08:19:12.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:12 smithi110 ceph-mon[28633]: monmap e1: 1 mons at {a=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:19:12.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:12 smithi110 ceph-mon[28633]: fsmap 2024-09-11T08:19:12.552 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:12 smithi110 ceph-mon[28633]: osdmap e1: 0 total, 0 up, 0 in 2024-09-11T08:19:12.552 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:12 smithi110 ceph-mon[28633]: mgrmap e1: no daemons active 2024-09-11T08:19:12.552 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:12 smithi110 ceph-mon[28633]: from='client.? 172.21.15.110:0/4052748416' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-11T08:19:12.554 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout cluster: 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout id: 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout services: 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.600637s) 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout data: 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout pgs: 2024-09-11T08:19:12.555 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:12.556 INFO:teuthology.orchestra.run.smithi110.stdout:mon is available 2024-09-11T08:19:12.556 INFO:teuthology.orchestra.run.smithi110.stdout:Assimilating anything we can from ceph.conf... 2024-09-11T08:19:13.511 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:13.511 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout [global] 2024-09-11T08:19:13.511 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout fsid = 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.110:3300,v1:172.21.15.110:6789] 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout [osd] 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-11T08:19:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-11T08:19:13.513 INFO:teuthology.orchestra.run.smithi110.stdout:Generating new minimal ceph.conf... 2024-09-11T08:19:13.616 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:13 smithi110 ceph-mon[28633]: from='client.? 172.21.15.110:0/1448323044' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-11T08:19:13.617 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:13 smithi110 ceph-mon[28633]: from='client.? 172.21.15.110:0/1448323044' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-11T08:19:14.489 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:14 smithi110 ceph-mon[28633]: from='client.? 172.21.15.110:0/2664068587' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:19:14.491 INFO:teuthology.orchestra.run.smithi110.stdout:Restarting the monitor... 2024-09-11T08:19:14.776 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:14 smithi110 systemd[1]: Stopping Ceph mon.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:19:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:14 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a[28629]: 2024-09-11T08:19:14.775+0000 7f43df683640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:19:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:14 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a[28629]: 2024-09-11T08:19:14.775+0000 7f43df683640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-11T08:19:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:14 smithi110 podman[28874]: 2024-09-11 08:19:14.908673005 +0000 UTC m=+0.253526721 container died 58cad7d83df78b52e96f429f3f89d21892c55b5923f617f1e744014109a4f32d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-11T08:19:15.620 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 podman[28874]: 2024-09-11 08:19:15.225234513 +0000 UTC m=+0.570088225 container cleanup 58cad7d83df78b52e96f429f3f89d21892c55b5923f617f1e744014109a4f32d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:19:15.620 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 bash[28874]: ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a 2024-09-11T08:19:15.620 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 podman[28886]: 2024-09-11 08:19:15.392953305 +0000 UTC m=+0.480659953 container remove 58cad7d83df78b52e96f429f3f89d21892c55b5923f617f1e744014109a4f32d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0) 2024-09-11T08:19:15.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-11T08:19:15.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 systemd[1]: Stopped Ceph mon.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:19:15.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 systemd[1]: Starting Ceph mon.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:19:15.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 podman[28978]: 2024-09-11 08:19:15.82712355 +0000 UTC m=+0.104739878 container create cf0c495a244ec64c4e0297d7d16f56794ea933956c7d81a96c7761cc264875e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-11T08:19:15.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:15 smithi110 podman[28978]: 2024-09-11 08:19:15.738327948 +0000 UTC m=+0.015944241 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:19:16.125 INFO:teuthology.orchestra.run.smithi110.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-11T08:19:16.239 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 podman[28978]: 2024-09-11 08:19:16.037694009 +0000 UTC m=+0.315310291 container init cf0c495a244ec64c4e0297d7d16f56794ea933956c7d81a96c7761cc264875e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-11T08:19:16.239 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 podman[28978]: 2024-09-11 08:19:16.040732 +0000 UTC m=+0.318348283 container start cf0c495a244ec64c4e0297d7d16f56794ea933956c7d81a96c7761cc264875e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-11T08:19:16.239 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: set uid:gid to 167:167 (ceph:ceph) 2024-09-11T08:19:16.239 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable), process ceph-mon, pid 2 2024-09-11T08:19:16.239 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: pidfile_write: ignore empty --pid-file 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: load: jerasure load: lrc 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: RocksDB version: 6.15.5 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Compile date Sep 10 2024 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: DB SUMMARY 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: DB Session ID: S3LKGTQGRMD3H8YW5TEX 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: CURRENT file: CURRENT 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: IDENTITY file: IDENTITY 2024-09-11T08:19:16.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 84170 ; 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.error_if_exists: 0 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.create_if_missing: 0 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.paranoid_checks: 1 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.env: 0x56505ce45440 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.fs: Posix File System 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.info_log: 0x56505d9d4ec0 2024-09-11T08:19:16.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_file_opening_threads: 16 2024-09-11T08:19:16.242 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.statistics: (nil) 2024-09-11T08:19:16.242 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.use_fsync: 0 2024-09-11T08:19:16.243 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_log_file_size: 0 2024-09-11T08:19:16.243 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.keep_log_file_num: 1000 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.recycle_log_file_num: 0 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.allow_fallocate: 1 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.allow_mmap_reads: 0 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.allow_mmap_writes: 0 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.use_direct_reads: 0 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.create_missing_column_families: 0 2024-09-11T08:19:16.244 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.db_log_dir: 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.advise_random_on_open: 1 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.db_write_buffer_size: 0 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.write_buffer_manager: 0x56505da92780 2024-09-11T08:19:16.245 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.rate_limiter: (nil) 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.wal_recovery_mode: 2 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.enable_thread_tracking: 0 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.enable_pipelined_write: 0 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.unordered_write: 0 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-11T08:19:16.246 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.row_cache: None 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.wal_filter: None 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.allow_ingest_behind: 0 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.preserve_deletes: 0 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.two_write_queues: 0 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.manual_wal_flush: 0 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.atomic_flush: 0 2024-09-11T08:19:16.247 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.log_readahead_size: 0 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.best_efforts_recovery: 0 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.allow_data_in_errors: 0 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.db_host_id: __hostname__ 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_background_jobs: 2 2024-09-11T08:19:16.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_background_compactions: -1 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_subcompactions: 1 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_total_wal_size: 0 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-11T08:19:16.249 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_open_files: -1 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bytes_per_sync: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_readahead_size: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_background_flushes: -1 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Compression algorithms supported: 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kZSTD supported: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kXpressCompression supported: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kBZip2Compression supported: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kLZ4Compression supported: 1 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kZlibCompression supported: 1 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kLZ4HCCompression supported: 1 2024-09-11T08:19:16.250 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: kSnappyCompression supported: 1 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.merge_operator: 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_filter: None 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_filter_factory: None 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.sst_partitioner_factory: None 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-11T08:19:16.251 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56505d97ba38) 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: cache_index_and_filter_blocks: 1 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: pin_top_level_index_and_filter: 1 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: index_type: 0 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: data_block_index_type: 0 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: index_shortening: 1 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: hash_index_allow_collision: 1 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: checksum: 1 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: no_block_cache: 0 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_cache: 0x56505d9b1350 2024-09-11T08:19:16.252 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_cache_name: BinnedLRUCache 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_cache_options: 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: capacity : 536870912 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: num_shard_bits : 4 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: strict_capacity_limit : 0 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: high_pri_pool_ratio: 0.000 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_cache_compressed: (nil) 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: persistent_cache: (nil) 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_size: 4096 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_size_deviation: 10 2024-09-11T08:19:16.253 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_restart_interval: 16 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: index_block_restart_interval: 1 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: metadata_block_size: 4096 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: partition_filters: 0 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: use_delta_encoding: 1 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: whole_key_filtering: 1 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: verify_compression: 0 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: read_amp_bytes_per_bit: 0 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: format_version: 4 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: enable_index_compression: 1 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout: block_align: 0 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.write_buffer_size: 33554432 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_write_buffer_number: 2 2024-09-11T08:19:16.254 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression: NoCompression 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression: Disabled 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.prefix_extractor: nullptr 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.num_levels: 7 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:19:16.255 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression_opts.level: 32767 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression_opts.strategy: 0 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compression_opts.enabled: false 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-11T08:19:16.256 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.target_file_size_base: 67108864 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-11T08:19:16.257 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.arena_block_size: 4194304 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.disable_auto_compactions: 0 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-11T08:19:16.258 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.inplace_update_support: 0 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.bloom_locality: 0 2024-09-11T08:19:16.259 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.max_successive_merges: 0 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.paranoid_file_checks: 0 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.force_consistency_checks: 1 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.report_bg_io_stats: 0 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.ttl: 2592000 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.enable_blob_files: false 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.min_blob_size: 0 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.blob_file_size: 268435456 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-11T08:19:16.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726042756063147, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726042756063862, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80757, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78996, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10080, "raw_average_key_size": 47, "raw_value_size": 73248, "raw_average_value_size": 343, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726042756, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "45113a30-89f3-4584-8095-b17839ede243", "db_session_id": "S3LKGTQGRMD3H8YW5TEX"}} 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726042756064226, "job": 1, "event": "recovery_finished"} 2024-09-11T08:19:16.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56505d9e4700 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: DB pointer 0x56505d9b4000 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: ** DB Stats ** 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:19:16.262 INFO:journalctl@ceph.mon.a.smithi110.stdout: 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: ** Compaction Stats [default] ** 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: L0 2/0 80.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.5 0.00 0.00 1 0.000 0 0 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: Sum 2/0 80.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.5 0.00 0.00 1 0.000 0 0 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.5 0.00 0.00 1 0.000 0 0 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: ** Compaction Stats [default] ** 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.5 0.00 0.00 1 0.000 0 0 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:19:16.263 INFO:journalctl@ceph.mon.a.smithi110.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: Cumulative compaction: 0.00 GB write, 25.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: Interval compaction: 0.00 GB write, 25.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: ** Compaction Stats [default] ** 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:19:16.264 INFO:journalctl@ceph.mon.a.smithi110.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: L0 2/0 80.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.5 0.00 0.00 1 0.000 0 0 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: Sum 2/0 80.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.5 0.00 0.00 1 0.000 0 0 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: ** Compaction Stats [default] ** 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.5 0.00 0.00 1 0.000 0 0 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:19:16.265 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: Cumulative compaction: 0.00 GB write, 25.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: starting mon.a rank 0 at public addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] at bind addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???) e1 preinit fsid 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???).mds e1 new map 2024-09-11T08:19:16.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???).mds e1 print_map 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout: e1 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout: legacy client fscid: -1 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout: 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout: No filesystems configured 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:19:16.267 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-11T08:19:16.268 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 bash[28978]: cf0c495a244ec64c4e0297d7d16f56794ea933956c7d81a96c7761cc264875e1 2024-09-11T08:19:16.268 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-11T08:19:16.268 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: monmap e1: 1 mons at {a=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:19:16.268 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: fsmap 2024-09-11T08:19:16.268 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: osdmap e1: 0 total, 0 up, 0 in 2024-09-11T08:19:16.268 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 ceph-mon[28998]: mgrmap e1: no daemons active 2024-09-11T08:19:16.268 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:16 smithi110 systemd[1]: Started Ceph mon.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:19:17.239 INFO:teuthology.orchestra.run.smithi110.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-11T08:19:17.240 INFO:teuthology.orchestra.run.smithi110.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:19:17.240 INFO:teuthology.orchestra.run.smithi110.stdout:Creating mgr... 2024-09-11T08:19:17.240 INFO:teuthology.orchestra.run.smithi110.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-11T08:19:17.241 INFO:teuthology.orchestra.run.smithi110.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-11T08:19:17.419 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y 2024-09-11T08:19:17.419 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service: Unit ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service not loaded. 2024-09-11T08:19:17.567 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2850e098-7016-11ef-bcea-c7b262605968.target.wants/ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-2850e098-7016-11ef-bcea-c7b262605968@.service. 2024-09-11T08:19:17.791 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:17 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/279667983' entity='client.admin' 2024-09-11T08:19:18.116 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 podman[29210]: 2024-09-11 08:19:18.018524373 +0000 UTC m=+0.340487513 container start 6dce3c0c4cd04a90db33a4b29b582d632f47c112e67a6bfc842922848c04e757 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-11T08:19:18.132 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:19:18.132 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout disabled 2024-09-11T08:19:18.141 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:19:18.141 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout inactive 2024-09-11T08:19:18.141 INFO:teuthology.orchestra.run.smithi110.stdout:firewalld.service is not enabled 2024-09-11T08:19:18.141 INFO:teuthology.orchestra.run.smithi110.stdout:Not possible to enable service . firewalld.service is not available 2024-09-11T08:19:18.152 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:19:18.152 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout disabled 2024-09-11T08:19:18.161 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:19:18.161 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout inactive 2024-09-11T08:19:18.161 INFO:teuthology.orchestra.run.smithi110.stdout:firewalld.service is not enabled 2024-09-11T08:19:18.161 INFO:teuthology.orchestra.run.smithi110.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-11T08:19:18.161 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for mgr to start... 2024-09-11T08:19:18.161 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for mgr... 2024-09-11T08:19:18.383 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 bash[29210]: 6dce3c0c4cd04a90db33a4b29b582d632f47c112e67a6bfc842922848c04e757 2024-09-11T08:19:18.383 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 systemd[1]: Started Ceph mgr.y for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:19:18.383 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:18.134+0000 7f8900ef5ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:19:18.383 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:18.199+0000 7f8900ef5ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:19:18.383 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:18.263+0000 7f8900ef5ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:19:18.644 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:18.390+0000 7f8900ef5ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:19:18.644 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:18.529+0000 7f8900ef5ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:19:18.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4243737321' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-11T08:19:18.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:18.799+0000 7f8900ef5ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:19:18.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:18 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:18.859+0000 7f8900ef5ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:19:19.206 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:19.206 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:19:19.206 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "fsid": "2850e098-7016-11ef-bcea-c7b262605968", 2024-09-11T08:19:19.206 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T08:19:19.206 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T08:19:19.206 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 0 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:19.207 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T08:19:19.208 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T08:19:19.209 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:19.210 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T08:19:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T08:19:11.379425+0000", 2024-09-11T08:19:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:19:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T08:19:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:19:19.211 INFO:teuthology.orchestra.run.smithi110.stdout:mgr not available, waiting (1/15)... 2024-09-11T08:19:19.303 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:19 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:19.024+0000 7f8900ef5ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:19:20.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:19 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:19.865+0000 7f8900ef5ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:19:20.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:19 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:19.928+0000 7f8900ef5ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:19:20.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:20 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:20.625+0000 7f8900ef5ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:19:20.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:20 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:20.734+0000 7f8900ef5ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:19:20.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:20 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:20.852+0000 7f8900ef5ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:19:21.163 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:20 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:20.910+0000 7f8900ef5ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:19:21.163 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.039+0000 7f8900ef5ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:19:21.163 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.105+0000 7f8900ef5ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:19:21.163 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.162+0000 7f8900ef5ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:19:21.436 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.305+0000 7f8900ef5ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:19:21.436 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.370+0000 7f8900ef5ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:19:21.436 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.435+0000 7f8900ef5ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:19:21.700 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.520+0000 7f8900ef5ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:19:21.700 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.583+0000 7f8900ef5ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:19:21.700 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:21.644+0000 7f8900ef5ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:19:22.046 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:21 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3652237652' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-11T08:19:22.291 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:22.291 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:19:22.291 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "fsid": "2850e098-7016-11ef-bcea-c7b262605968", 2024-09-11T08:19:22.291 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T08:19:22.291 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T08:19:22.291 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 0 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-11T08:19:22.292 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T08:19:22.293 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T08:19:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T08:19:11.379425+0000", 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:19:22.295 INFO:teuthology.orchestra.run.smithi110.stdout:mgr not available, waiting (2/15)... 2024-09-11T08:19:22.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:22.397+0000 7f8900ef5ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:19:23.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: Activating manager daemon y 2024-09-11T08:19:23.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: mgrmap e2: y(active, starting, since 0.00296393s) 2024-09-11T08:19:23.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:19:23.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: Manager daemon y is now available 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' 2024-09-11T08:19:23.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:22 smithi110 ceph-mon[28998]: from='mgr.14100 172.21.15.110:0/1511827069' entity='mgr.y' 2024-09-11T08:19:24.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:24 smithi110 ceph-mon[28998]: mgrmap e3: y(active, since 1.00689s) 2024-09-11T08:19:25.533 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:25.533 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:19:25.533 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "fsid": "2850e098-7016-11ef-bcea-c7b262605968", 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "health": { 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 0 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "a" 2024-09-11T08:19:25.534 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-11T08:19:25.535 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-11T08:19:25.539 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-11T08:19:25.540 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "restful" 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ], 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "modified": "2024-09-11T08:19:11.379425+0000", 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout }, 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:19:25.541 INFO:teuthology.orchestra.run.smithi110.stdout:mgr is available 2024-09-11T08:19:25.656 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:25 smithi110 ceph-mon[28998]: mgrmap e4: y(active, since 2s) 2024-09-11T08:19:25.657 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:25 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3810117868' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-11T08:19:29.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:28 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3084433549' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-11T08:19:35.525 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:35.525 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout [global] 2024-09-11T08:19:35.525 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout fsid = 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:35.525 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-11T08:19:35.525 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-11T08:19:35.525 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-11T08:19:35.525 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout [osd] 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-11T08:19:35.526 INFO:teuthology.orchestra.run.smithi110.stdout:Enabling cephadm module... 2024-09-11T08:19:36.439 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4160389677' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-11T08:19:36.439 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setuser ceph since I am not root 2024-09-11T08:19:36.439 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setgroup ceph since I am not root 2024-09-11T08:19:36.440 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:36.281+0000 7fbeb322bec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:19:36.440 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:36.334+0000 7fbeb322bec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:19:36.440 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:36.402+0000 7fbeb322bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:19:36.698 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:36.525+0000 7fbeb322bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:19:36.698 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:36.661+0000 7fbeb322bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:19:36.976 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:36.929+0000 7fbeb322bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:19:37.260 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:37 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4160389677' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-11T08:19:37.261 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:37 smithi110 ceph-mon[28998]: mgrmap e5: y(active, since 13s) 2024-09-11T08:19:37.261 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:36.990+0000 7fbeb322bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:19:37.261 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:37.155+0000 7fbeb322bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:19:37.808 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:19:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-11T08:19:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T08:19:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-11T08:19:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-11T08:19:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:19:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for the mgr to restart... 2024-09-11T08:19:37.809 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for mgr epoch 5... 2024-09-11T08:19:38.079 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:38.012+0000 7fbeb322bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:19:38.079 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:38.077+0000 7fbeb322bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:19:38.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:38 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3562921427' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-11T08:19:39.074 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:38.784+0000 7fbeb322bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:19:39.074 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:38.897+0000 7fbeb322bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:19:39.075 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.014+0000 7fbeb322bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:19:39.075 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.072+0000 7fbeb322bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:19:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.196+0000 7fbeb322bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:19:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.261+0000 7fbeb322bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:19:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.319+0000 7fbeb322bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:19:39.718 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.452+0000 7fbeb322bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:19:39.718 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.514+0000 7fbeb322bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:19:39.718 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.578+0000 7fbeb322bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:19:39.718 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.660+0000 7fbeb322bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:19:39.718 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.716+0000 7fbeb322bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:19:40.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:39.772+0000 7fbeb322bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:19:40.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:40.503+0000 7fbeb322bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:19:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: Active manager daemon y restarted 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: Activating manager daemon y 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: osdmap e2: 0 total, 0 up, 0 in 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: mgrmap e6: y(active, starting, since 0.00302274s) 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: Manager daemon y is now available 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:40.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:40.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:40.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:40.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:19:40.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:40 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:19:41.803 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:41 smithi110 ceph-mon[28998]: Found migration_current of "None". Setting to last migration. 2024-09-11T08:19:41.803 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:41 smithi110 ceph-mon[28998]: mgrmap e7: y(active, since 1.00479s) 2024-09-11T08:19:42.073 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:19:42.073 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-11T08:19:42.073 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-11T08:19:42.073 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:19:42.073 INFO:teuthology.orchestra.run.smithi110.stdout:mgr epoch 5 is available 2024-09-11T08:19:42.074 INFO:teuthology.orchestra.run.smithi110.stdout:Setting orchestrator backend to cephadm... 2024-09-11T08:19:42.652 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-mon[28998]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-11T08:19:42.652 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-mon[28998]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:42.925+0000 7fbe655d9640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:19:42] ENGINE Error in HTTPServer.serve 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: Traceback (most recent call last): 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._connections.run(self.expiration_interval) 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._run(expiration_interval) 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:19:43.291 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: s = self.context.wrap_socket( 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: return self.sslsocket_class._create( 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self.do_handshake() 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._sslobj.do_handshake() 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:19:43.292 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:43.899 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: mgrmap e8: y(active, since 2s) 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: [11/Sep/2024:08:19:42] ENGINE Bus STARTING 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: [11/Sep/2024:08:19:42] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: [11/Sep/2024:08:19:42] ENGINE Bus STARTED 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: [11/Sep/2024:08:19:42] ENGINE Error in HTTPServer.serve 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:19:43.900 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: self.do_handshake() 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:19:43.901 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:19:43.902 INFO:journalctl@ceph.mon.a.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:19:43.902 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:43.902 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:43 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:44.320 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-11T08:19:44.320 INFO:teuthology.orchestra.run.smithi110.stdout:Generating ssh key... 2024-09-11T08:19:44.604 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:44 smithi110 ceph-mon[28998]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:19:45.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: Generating public/private rsa key pair. 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: Your identification has been saved in /tmp/tmp1py_j1zi/key 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: Your public key has been saved in /tmp/tmp1py_j1zi/key.pub 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: The key fingerprint is: 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: SHA256:brzf3hp665MsHWqQ7p99WGL7YLe220yVzEzBAU32B50 ceph-2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: The key's randomart image is: 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: +---[RSA 3072]----+ 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | .**o| 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | .E+| 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | .o| 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | = o| 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | S. =.| 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | oo + . .| 2024-09-11T08:19:45.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | .+. =+B. .| 2024-09-11T08:19:45.405 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | ...+*O=o= | 2024-09-11T08:19:45.405 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: | .o+==BB*oo| 2024-09-11T08:19:45.405 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: +----[SHA256]-----+ 2024-09-11T08:19:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:46 smithi110 ceph-mon[28998]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:19:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:46 smithi110 ceph-mon[28998]: Generating ssh key... 2024-09-11T08:19:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:46 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:46.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:46 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:46.800 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzpwELSAdVzUotJioBq11MCuxSegZFVGZIhM4IY/Yi2CL57TbybzW7eLxV5MiOC9YYDyoYyIZYxXnbrcKFs8iSMDvWapiu+DJGHrd7bRqaz5avrT/J0ItS95qSQ8FMBeSKUxzkZ0DecsYHk/a3z2jz1t3v+y4RaKgjcgEw95lPu+XADjc8M/NXWExeDauV2YSyWnuNB2dTPEYp4q/3iWK7J28p1JRtvTITM53Ta3CnE2UDfHssD3qlEFVaUfBKMNVO1HF+1PnFQpFm8//hygmTCWCY/RF1Dg1WTHnBP6XdvWTsX3xYhLYo5VOudrxfOU5i/QYEqZr7PFuwn5VWzfHHM/MHHrCysnRft70oce8oMiTL70QMzsl9cY12xGp8MlNNq+1hsmxy2zK2YYknqWVslOfsygEtAsKMA8+CCrHfN3RG9FL4xCUqe1FSjDFsuPQvTKtuaxd91e1WWqqu4NWfoK10FAXKgdrC6LwlpbynBlH+mu9SuIP/JnE/LkRjGzE= ceph-2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:19:46.800 INFO:teuthology.orchestra.run.smithi110.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-11T08:19:46.800 INFO:teuthology.orchestra.run.smithi110.stdout:Adding key to root@localhost authorized_keys... 2024-09-11T08:19:46.810 INFO:teuthology.orchestra.run.smithi110.stdout:Adding host smithi110... 2024-09-11T08:19:47.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:47 smithi110 ceph-mon[28998]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:19:48.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:48 smithi110 ceph-mon[28998]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "addr": "172.21.15.110", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:19:49.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:49 smithi110 ceph-mon[28998]: Deploying cephadm binary to smithi110 2024-09-11T08:19:50.625 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout Added host 'smithi110' with addr '172.21.15.110' 2024-09-11T08:19:50.625 INFO:teuthology.orchestra.run.smithi110.stdout:Deploying unmanaged mon service... 2024-09-11T08:19:51.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:51 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:51.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:51 smithi110 ceph-mon[28998]: Added host smithi110 2024-09-11T08:19:51.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:51 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:19:51.759 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-11T08:19:51.759 INFO:teuthology.orchestra.run.smithi110.stdout:Deploying unmanaged mgr service... 2024-09-11T08:19:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:52 smithi110 ceph-mon[28998]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:19:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:52 smithi110 ceph-mon[28998]: Saving service mon spec with placement count:5 2024-09-11T08:19:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:52 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:52.875 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-11T08:19:53.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:53 smithi110 ceph-mon[28998]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:19:53.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:53 smithi110 ceph-mon[28998]: Saving service mgr spec with placement count:2 2024-09-11T08:19:53.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:53 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:54.594 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:54 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2985040960' entity='client.admin' 2024-09-11T08:19:54.594 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:54 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:56.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:55 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:56.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:55 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1987141767' entity='client.admin' 2024-09-11T08:19:56.640 INFO:teuthology.orchestra.run.smithi110.stdout:Enabling the dashboard module... 2024-09-11T08:19:57.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:57 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:57.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:57 smithi110 ceph-mon[28998]: from='mgr.14118 172.21.15.110:0/498642802' entity='mgr.y' 2024-09-11T08:19:58.343 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1065529573' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-11T08:19:58.343 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setuser ceph since I am not root 2024-09-11T08:19:58.343 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setgroup ceph since I am not root 2024-09-11T08:19:58.343 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:58.221+0000 7f26e9a0bec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:19:58.343 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:58.276+0000 7f26e9a0bec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:19:58.344 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:58.342+0000 7f26e9a0bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:19:58.604 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:58.466+0000 7f26e9a0bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:19:58.605 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:58.604+0000 7f26e9a0bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:19:59.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:59 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1065529573' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-11T08:19:59.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:19:59 smithi110 ceph-mon[28998]: mgrmap e9: y(active, since 17s) 2024-09-11T08:19:59.154 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:58.866+0000 7f26e9a0bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:19:59.154 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:58 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:58.926+0000 7f26e9a0bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:19:59.154 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:59 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:59.093+0000 7f26e9a0bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for the mgr to restart... 2024-09-11T08:19:59.929 INFO:teuthology.orchestra.run.smithi110.stdout:Waiting for mgr epoch 9... 2024-09-11T08:20:00.190 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:00 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2063273120' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-11T08:20:00.190 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:19:59 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:19:59.960+0000 7f26e9a0bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:20:00.190 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:00 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:00.025+0000 7f26e9a0bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:20:01.033 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:00 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:00.743+0000 7f26e9a0bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:20:01.033 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:00 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:00.853+0000 7f26e9a0bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:20:01.033 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:00 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:00.973+0000 7f26e9a0bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:20:01.033 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.032+0000 7f26e9a0bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:20:01.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.158+0000 7f26e9a0bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:20:01.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.223+0000 7f26e9a0bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:20:01.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.280+0000 7f26e9a0bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:20:01.681 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.414+0000 7f26e9a0bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:20:01.681 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.477+0000 7f26e9a0bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:20:01.681 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.541+0000 7f26e9a0bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:20:01.682 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.623+0000 7f26e9a0bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:20:01.682 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.680+0000 7f26e9a0bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:20:02.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:01.737+0000 7f26e9a0bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:20:02.673 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:02.493+0000 7f26e9a0bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: Active manager daemon y restarted 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: Activating manager daemon y 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: osdmap e3: 0 total, 0 up, 0 in 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: mgrmap e10: y(active, starting, since 0.00337613s) 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: Manager daemon y is now available 2024-09-11T08:20:02.674 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:02.675 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:20:02.675 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:20:03.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:03.648+0000 7f269a568640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:20:03] ENGINE Error in HTTPServer.serve 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: Traceback (most recent call last): 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._connections.run(self.expiration_interval) 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._run(expiration_interval) 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:20:03.904 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: s = self.context.wrap_socket( 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: return self.sslsocket_class._create( 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self.do_handshake() 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._sslobj.do_handshake() 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:20:03.905 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:03 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:04.231 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout { 2024-09-11T08:20:04.232 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-11T08:20:04.232 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-11T08:20:04.232 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout } 2024-09-11T08:20:04.232 INFO:teuthology.orchestra.run.smithi110.stdout:mgr epoch 9 is available 2024-09-11T08:20:04.232 INFO:teuthology.orchestra.run.smithi110.stdout:Generating a dashboard self-signed certificate... 2024-09-11T08:20:04.599 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:04 smithi110 ceph-mon[28998]: mgrmap e11: y(active, since 1.00696s) 2024-09-11T08:20:04.599 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:04 smithi110 ceph-mon[28998]: [11/Sep/2024:08:20:03] ENGINE Bus STARTING 2024-09-11T08:20:04.599 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:04 smithi110 ceph-mon[28998]: [11/Sep/2024:08:20:03] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:20:04.599 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:04 smithi110 ceph-mon[28998]: [11/Sep/2024:08:20:03] ENGINE Bus STARTED 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:04 smithi110 ceph-mon[28998]: [11/Sep/2024:08:20:03] ENGINE Error in HTTPServer.serve 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:20:04.600 INFO:journalctl@ceph.mon.a.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout: self.do_handshake() 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:04.601 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:05.454 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-11T08:20:05.455 INFO:teuthology.orchestra.run.smithi110.stdout:Creating initial admin user... 2024-09-11T08:20:05.889 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:05 smithi110 ceph-mon[28998]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:05.889 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:05 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:05.889 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:05 smithi110 ceph-mon[28998]: mgrmap e12: y(active, since 2s) 2024-09-11T08:20:05.889 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:05 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:05.889 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:05 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:06.957 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$sHJL/..CNRnqOKA0a0cFy.TT9vGxmADebjW.XvcaiVeyGDyqe0tl.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726042806, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-11T08:20:06.957 INFO:teuthology.orchestra.run.smithi110.stdout:Fetching dashboard port number... 2024-09-11T08:20:07.072 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:06 smithi110 ceph-mon[28998]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:07.072 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:06 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:08.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:07 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3468777761' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-11T08:20:08.425 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stdout 8443 2024-09-11T08:20:08.435 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-11T08:20:08.435 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout disabled 2024-09-11T08:20:08.443 INFO:teuthology.orchestra.run.smithi110.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-11T08:20:08.443 INFO:teuthology.orchestra.run.smithi110.stdout:systemctl: stdout inactive 2024-09-11T08:20:08.443 INFO:teuthology.orchestra.run.smithi110.stdout:firewalld.service is not enabled 2024-09-11T08:20:08.443 INFO:teuthology.orchestra.run.smithi110.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-11T08:20:08.445 INFO:teuthology.orchestra.run.smithi110.stdout:Ceph Dashboard is now available at: 2024-09-11T08:20:08.445 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:08.445 INFO:teuthology.orchestra.run.smithi110.stdout: URL: https://smithi110.front.sepia.ceph.com:8443/ 2024-09-11T08:20:08.445 INFO:teuthology.orchestra.run.smithi110.stdout: User: admin 2024-09-11T08:20:08.445 INFO:teuthology.orchestra.run.smithi110.stdout: Password: fknqr4twc4 2024-09-11T08:20:08.445 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:08.446 INFO:teuthology.orchestra.run.smithi110.stdout:Saving cluster configuration to /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config directory 2024-09-11T08:20:08.446 INFO:teuthology.orchestra.run.smithi110.stdout:Enabling autotune for osd_memory_target 2024-09-11T08:20:09.884 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:09 smithi110 ceph-mon[28998]: mgrmap e13: y(active, since 6s) 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout:Or, if you are only running a single cluster on this host: 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.641 INFO:teuthology.orchestra.run.smithi110.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout: ceph telemetry on 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout:For more information see: 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:10.642 INFO:teuthology.orchestra.run.smithi110.stdout:Bootstrap complete. 2024-09-11T08:20:10.677 INFO:tasks.cephadm:Fetching config... 2024-09-11T08:20:10.677 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:20:10.677 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-11T08:20:10.693 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-11T08:20:10.693 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:20:10.693 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-11T08:20:10.749 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-11T08:20:10.749 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:20:10.749 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-11T08:20:10.813 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-11T08:20:10.813 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:20:10.813 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-11T08:20:10.869 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-11T08:20:10.869 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzpwELSAdVzUotJioBq11MCuxSegZFVGZIhM4IY/Yi2CL57TbybzW7eLxV5MiOC9YYDyoYyIZYxXnbrcKFs8iSMDvWapiu+DJGHrd7bRqaz5avrT/J0ItS95qSQ8FMBeSKUxzkZ0DecsYHk/a3z2jz1t3v+y4RaKgjcgEw95lPu+XADjc8M/NXWExeDauV2YSyWnuNB2dTPEYp4q/3iWK7J28p1JRtvTITM53Ta3CnE2UDfHssD3qlEFVaUfBKMNVO1HF+1PnFQpFm8//hygmTCWCY/RF1Dg1WTHnBP6XdvWTsX3xYhLYo5VOudrxfOU5i/QYEqZr7PFuwn5VWzfHHM/MHHrCysnRft70oce8oMiTL70QMzsl9cY12xGp8MlNNq+1hsmxy2zK2YYknqWVslOfsygEtAsKMA8+CCrHfN3RG9FL4xCUqe1FSjDFsuPQvTKtuaxd91e1WWqqu4NWfoK10FAXKgdrC6LwlpbynBlH+mu9SuIP/JnE/LkRjGzE= ceph-2850e098-7016-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-11T08:20:10.950 INFO:teuthology.orchestra.run.smithi110.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzpwELSAdVzUotJioBq11MCuxSegZFVGZIhM4IY/Yi2CL57TbybzW7eLxV5MiOC9YYDyoYyIZYxXnbrcKFs8iSMDvWapiu+DJGHrd7bRqaz5avrT/J0ItS95qSQ8FMBeSKUxzkZ0DecsYHk/a3z2jz1t3v+y4RaKgjcgEw95lPu+XADjc8M/NXWExeDauV2YSyWnuNB2dTPEYp4q/3iWK7J28p1JRtvTITM53Ta3CnE2UDfHssD3qlEFVaUfBKMNVO1HF+1PnFQpFm8//hygmTCWCY/RF1Dg1WTHnBP6XdvWTsX3xYhLYo5VOudrxfOU5i/QYEqZr7PFuwn5VWzfHHM/MHHrCysnRft70oce8oMiTL70QMzsl9cY12xGp8MlNNq+1hsmxy2zK2YYknqWVslOfsygEtAsKMA8+CCrHfN3RG9FL4xCUqe1FSjDFsuPQvTKtuaxd91e1WWqqu4NWfoK10FAXKgdrC6LwlpbynBlH+mu9SuIP/JnE/LkRjGzE= ceph-2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:20:10.979 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzpwELSAdVzUotJioBq11MCuxSegZFVGZIhM4IY/Yi2CL57TbybzW7eLxV5MiOC9YYDyoYyIZYxXnbrcKFs8iSMDvWapiu+DJGHrd7bRqaz5avrT/J0ItS95qSQ8FMBeSKUxzkZ0DecsYHk/a3z2jz1t3v+y4RaKgjcgEw95lPu+XADjc8M/NXWExeDauV2YSyWnuNB2dTPEYp4q/3iWK7J28p1JRtvTITM53Ta3CnE2UDfHssD3qlEFVaUfBKMNVO1HF+1PnFQpFm8//hygmTCWCY/RF1Dg1WTHnBP6XdvWTsX3xYhLYo5VOudrxfOU5i/QYEqZr7PFuwn5VWzfHHM/MHHrCysnRft70oce8oMiTL70QMzsl9cY12xGp8MlNNq+1hsmxy2zK2YYknqWVslOfsygEtAsKMA8+CCrHfN3RG9FL4xCUqe1FSjDFsuPQvTKtuaxd91e1WWqqu4NWfoK10FAXKgdrC6LwlpbynBlH+mu9SuIP/JnE/LkRjGzE= ceph-2850e098-7016-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-11T08:20:11.098 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzpwELSAdVzUotJioBq11MCuxSegZFVGZIhM4IY/Yi2CL57TbybzW7eLxV5MiOC9YYDyoYyIZYxXnbrcKFs8iSMDvWapiu+DJGHrd7bRqaz5avrT/J0ItS95qSQ8FMBeSKUxzkZ0DecsYHk/a3z2jz1t3v+y4RaKgjcgEw95lPu+XADjc8M/NXWExeDauV2YSyWnuNB2dTPEYp4q/3iWK7J28p1JRtvTITM53Ta3CnE2UDfHssD3qlEFVaUfBKMNVO1HF+1PnFQpFm8//hygmTCWCY/RF1Dg1WTHnBP6XdvWTsX3xYhLYo5VOudrxfOU5i/QYEqZr7PFuwn5VWzfHHM/MHHrCysnRft70oce8oMiTL70QMzsl9cY12xGp8MlNNq+1hsmxy2zK2YYknqWVslOfsygEtAsKMA8+CCrHfN3RG9FL4xCUqe1FSjDFsuPQvTKtuaxd91e1WWqqu4NWfoK10FAXKgdrC6LwlpbynBlH+mu9SuIP/JnE/LkRjGzE= ceph-2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:20:11.111 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-11T08:20:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:11 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3634274826' entity='client.admin' 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3273271841' entity='client.admin' 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:15.269 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:15.293 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-11T08:20:15.294 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-11T08:20:15.529 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:17.843 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi139 2024-09-11T08:20:17.844 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:20:17.844 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.conf 2024-09-11T08:20:17.863 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:20:17.863 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:20:17.921 INFO:tasks.cephadm:Adding host smithi139 to orchestrator... 2024-09-11T08:20:17.921 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch host add smithi139 2024-09-11T08:20:18.160 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:18.280 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:18.280 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:18.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:18.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:18.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:18.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:20:18.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:18.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:18.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:19.341 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:19 smithi110 ceph-mon[28998]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:19.341 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:19 smithi110 ceph-mon[28998]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:20:19.341 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:19 smithi110 ceph-mon[28998]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:20:20.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:20 smithi110 ceph-mon[28998]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:21.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:21 smithi110 ceph-mon[28998]: Deploying cephadm binary to smithi139 2024-09-11T08:20:22.485 INFO:teuthology.orchestra.run.smithi110.stdout:Added host 'smithi139' with addr '172.21.15.139' 2024-09-11T08:20:23.038 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-11T08:20:23.276 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:23.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:23 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:23.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:23 smithi110 ceph-mon[28998]: Added host smithi139 2024-09-11T08:20:23.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:23 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:23.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:23 smithi110 ceph-mon[28998]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:23.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:23 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:23.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:23 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:24.904 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:20:24.904 INFO:teuthology.orchestra.run.smithi110.stdout:[{"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}] 2024-09-11T08:20:25.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:25.493 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-11T08:20:25.493 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-11T08:20:25.726 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:26.122 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:25 smithi110 ceph-mon[28998]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:26.123 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:25 smithi110 ceph-mon[28998]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:20:27.832 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2024-09-11T08:20:27.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/906337129' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-11T08:20:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:27.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:27.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:27 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:28.475 INFO:tasks.cephadm:Adding mon.a on smithi110 2024-09-11T08:20:28.475 INFO:tasks.cephadm:Adding mon.c on smithi110 2024-09-11T08:20:28.475 INFO:tasks.cephadm:Adding mon.b on smithi139 2024-09-11T08:20:28.476 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi110:172.21.15.110=a;smithi110:[v2:172.21.15.110:3301,v1:172.21.15.110:6790]=c;smithi139:172.21.15.139=b' 2024-09-11T08:20:28.717 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:29.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:28 smithi110 ceph-mon[28998]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:20:29.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:28 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/906337129' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-11T08:20:29.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:28 smithi110 ceph-mon[28998]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:20:30.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:29 smithi110 ceph-mon[28998]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:30.250 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled mon update... 2024-09-11T08:20:30.856 DEBUG:teuthology.orchestra.run.smithi110:mon.c> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.c.service 2024-09-11T08:20:30.859 DEBUG:teuthology.orchestra.run.smithi139:mon.b> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.b.service 2024-09-11T08:20:30.861 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T08:20:30.861 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T08:20:31.505 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:31.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi110:172.21.15.110=a;smithi110:[v2:172.21.15.110:3301,v1:172.21.15.110:6790]=c;smithi139:172.21.15.139=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: Saving service mon spec with placement smithi110:172.21.15.110=a;smithi110:[v2:172.21.15.110:3301,v1:172.21.15.110:6790]=c;smithi139:172.21.15.139=b;count:3 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:31.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:31.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: Deploying daemon mon.b on smithi139 2024-09-11T08:20:31.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:31 smithi110 ceph-mon[28998]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:33.075 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:20:33.075 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":1,"fsid":"2850e098-7016-11ef-bcea-c7b262605968","modified":"2024-09-11T08:19:08.446552Z","created":"2024-09-11T08:19:08.446552Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-11T08:20:33.076 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 1 2024-09-11T08:20:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:33 smithi110 ceph-mon[28998]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:33 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/1728607989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:20:34.691 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T08:20:34.691 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T08:20:34.955 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:35.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:35 smithi110 ceph-mon[28998]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:35.947 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:35 smithi139 systemd[1]: Starting Ceph mon.b for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:20:36.880 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:20:36.880 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":1,"fsid":"2850e098-7016-11ef-bcea-c7b262605968","modified":"2024-09-11T08:19:08.446552Z","created":"2024-09-11T08:19:08.446552Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-11T08:20:36.881 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 1 2024-09-11T08:20:36.886 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:36 smithi139 podman[32548]: 2024-09-11 08:20:36.680442972 +0000 UTC m=+0.202018651 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:20:36.886 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:36 smithi139 podman[32548]: 2024-09-11 08:20:36.784383717 +0000 UTC m=+0.305959375 container create 648681997ea9acf3e3afaed75b4053b282bbf59666a93cd905b8f01553fdb718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-11T08:20:37.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:36 smithi139 podman[32548]: 2024-09-11 08:20:36.995654433 +0000 UTC m=+0.517230097 container init 648681997ea9acf3e3afaed75b4053b282bbf59666a93cd905b8f01553fdb718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-11T08:20:37.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 podman[32548]: 2024-09-11 08:20:37.000648106 +0000 UTC m=+0.522223766 container start 648681997ea9acf3e3afaed75b4053b282bbf59666a93cd905b8f01553fdb718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: set uid:gid to 167:167 (ceph:ceph) 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: ceph version 17.2.7-1623-gd0e6828a (d0e6828a2016d48cf25ad84064e50742bb1c39b9) quincy (stable), process ceph-mon, pid 2 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: pidfile_write: ignore empty --pid-file 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: load: jerasure load: lrc 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: RocksDB version: 6.15.5 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Compile date Sep 10 2024 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: DB SUMMARY 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: DB Session ID: 0FCUADDLL24OEZKBMYKM 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: CURRENT file: CURRENT 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: IDENTITY file: IDENTITY 2024-09-11T08:20:37.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.error_if_exists: 0 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.create_if_missing: 0 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.paranoid_checks: 1 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.env: 0x55577deed440 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.fs: Posix File System 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.info_log: 0x55577f400ec0 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_file_opening_threads: 16 2024-09-11T08:20:37.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.statistics: (nil) 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.use_fsync: 0 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_log_file_size: 0 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.keep_log_file_num: 1000 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.recycle_log_file_num: 0 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.allow_fallocate: 1 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.allow_mmap_reads: 0 2024-09-11T08:20:37.326 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.allow_mmap_writes: 0 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.use_direct_reads: 0 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.create_missing_column_families: 0 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.db_log_dir: 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.advise_random_on_open: 1 2024-09-11T08:20:37.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.db_write_buffer_size: 0 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.write_buffer_manager: 0x55577f4be780 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.rate_limiter: (nil) 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.wal_recovery_mode: 2 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.enable_thread_tracking: 0 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.enable_pipelined_write: 0 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.unordered_write: 0 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-11T08:20:37.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.row_cache: None 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.wal_filter: None 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.allow_ingest_behind: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.preserve_deletes: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.two_write_queues: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.manual_wal_flush: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.atomic_flush: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-11T08:20:37.329 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.log_readahead_size: 0 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.best_efforts_recovery: 0 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.allow_data_in_errors: 0 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.db_host_id: __hostname__ 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_background_jobs: 2 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_background_compactions: -1 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_subcompactions: 1 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-11T08:20:37.330 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_total_wal_size: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_open_files: -1 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bytes_per_sync: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_readahead_size: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_background_flushes: -1 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Compression algorithms supported: 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kZSTD supported: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kXpressCompression supported: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kBZip2Compression supported: 0 2024-09-11T08:20:37.331 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kLZ4Compression supported: 1 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kZlibCompression supported: 1 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kLZ4HCCompression supported: 1 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: kSnappyCompression supported: 1 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.merge_operator: 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_filter: None 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_filter_factory: None 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.sst_partitioner_factory: None 2024-09-11T08:20:37.332 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55577f3a7a38) 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: cache_index_and_filter_blocks: 1 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: pin_top_level_index_and_filter: 1 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: index_type: 0 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: data_block_index_type: 0 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: index_shortening: 1 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: hash_index_allow_collision: 1 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: checksum: 1 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: no_block_cache: 0 2024-09-11T08:20:37.333 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache: 0x55577f3dd350 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache_name: BinnedLRUCache 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache_options: 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: capacity : 536870912 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: num_shard_bits : 4 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: strict_capacity_limit : 0 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: high_pri_pool_ratio: 0.000 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_cache_compressed: (nil) 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: persistent_cache: (nil) 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_size: 4096 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_size_deviation: 10 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_restart_interval: 16 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: index_block_restart_interval: 1 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: metadata_block_size: 4096 2024-09-11T08:20:37.334 INFO:journalctl@ceph.mon.b.smithi139.stdout: partition_filters: 0 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: use_delta_encoding: 1 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: whole_key_filtering: 1 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: verify_compression: 0 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: read_amp_bytes_per_bit: 0 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: format_version: 4 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: enable_index_compression: 1 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout: block_align: 0 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.write_buffer_size: 33554432 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_write_buffer_number: 2 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression: NoCompression 2024-09-11T08:20:37.335 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression: Disabled 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.prefix_extractor: nullptr 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.num_levels: 7 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-11T08:20:37.336 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression_opts.level: 32767 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression_opts.strategy: 0 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compression_opts.enabled: false 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.target_file_size_base: 67108864 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-11T08:20:37.337 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.arena_block_size: 4194304 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-11T08:20:37.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.disable_auto_compactions: 0 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.inplace_update_support: 0 2024-09-11T08:20:37.339 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.bloom_locality: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.max_successive_merges: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.paranoid_file_checks: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.force_consistency_checks: 1 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.report_bg_io_stats: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.ttl: 2592000 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.enable_blob_files: false 2024-09-11T08:20:37.340 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.min_blob_size: 0 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.blob_file_size: 268435456 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726042837024953, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726042837025255, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726042837, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "99a219b8-f387-476a-81de-b69d38226c51", "db_session_id": "0FCUADDLL24OEZKBMYKM"}} 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-11T08:20:37.341 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726042837025621, "job": 1, "event": "recovery_finished"} 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55577f410700 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: DB pointer 0x55577f3e0000 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** DB Stats ** 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-11T08:20:37.342 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** Compaction Stats [default] ** 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** Compaction Stats [default] ** 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-11T08:20:37.343 INFO:journalctl@ceph.mon.b.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** Compaction Stats [default] ** 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:20:37.344 INFO:journalctl@ceph.mon.b.smithi139.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** Compaction Stats [default] ** 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-11T08:20:37.345 INFO:journalctl@ceph.mon.b.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: using public_addr v2:172.21.15.139:0/0 -> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: starting mon.b rank -1 at public addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] at bind addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(???) e0 preinit fsid 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).mds e1 new map 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: e1 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-11T08:20:37.346 INFO:journalctl@ceph.mon.b.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout: legacy client fscid: -1 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout: 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout: No filesystems configured 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:37.347 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi110:172.21.15.110=a;smithi110:[v2:172.21.15.110:3301,v1:172.21.15.110:6790]=c;smithi139:172.21.15.139=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: Saving service mon spec with placement smithi110:172.21.15.110=a;smithi110:[v2:172.21.15.110:3301,v1:172.21.15.110:6790]=c;smithi139:172.21.15.139=b;count:3 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: Deploying daemon mon.b on smithi139 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/1728607989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:37.348 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 ceph-mon[32601]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-11T08:20:37.349 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 bash[32548]: 648681997ea9acf3e3afaed75b4053b282bbf59666a93cd905b8f01553fdb718 2024-09-11T08:20:37.349 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:37 smithi139 systemd[1]: Started Ceph mon.b for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:20:38.673 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T08:20:38.674 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T08:20:38.941 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:42.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: mon.a calling monitor election 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: mon.b calling monitor election 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: monmap e2: 2 mons at {a=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],b=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: fsmap 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: mgrmap e13: y(active, since 39s) 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: overall HEALTH_OK 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:42.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: mon.a calling monitor election 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: mon.b calling monitor election 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: monmap e2: 2 mons at {a=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],b=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: fsmap 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: mgrmap e13: y(active, since 39s) 2024-09-11T08:20:42.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: overall HEALTH_OK 2024-09-11T08:20:42.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:42.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:42.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:42.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:42.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:43.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:43 smithi110 ceph-mon[28998]: Deploying daemon mon.c on smithi110 2024-09-11T08:20:43.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:43 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:43.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:43 smithi139 ceph-mon[32601]: Deploying daemon mon.c on smithi110 2024-09-11T08:20:43.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:43 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:43.584 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:20:43.584 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":2,"fsid":"2850e098-7016-11ef-bcea-c7b262605968","modified":"2024-09-11T08:20:37.049199Z","created":"2024-09-11T08:19:08.446552Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-11T08:20:43.585 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 2 2024-09-11T08:20:44.171 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:44 smithi139 ceph-mon[32601]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:44.172 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:44 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/1838177584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:20:44.213 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:44 smithi110 ceph-mon[28998]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:44.214 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:44 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/1838177584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:20:45.196 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-11T08:20:45.196 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-11T08:20:45.445 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:50.314 INFO:teuthology.orchestra.run.smithi139.stdout: 2024-09-11T08:20:50.314 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":3,"fsid":"2850e098-7016-11ef-bcea-c7b262605968","modified":"2024-09-11T08:20:45.023267Z","created":"2024-09-11T08:19:08.446552Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-11T08:20:50.315 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 3 2024-09-11T08:20:50.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: mon.a calling monitor election 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: mon.b calling monitor election 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: mon.c calling monitor election 2024-09-11T08:20:50.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: monmap e3: 3 mons at {a=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],b=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],c=[v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: fsmap 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: mgrmap e13: y(active, since 47s) 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: overall HEALTH_OK 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:50.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:20:50.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: mon.a calling monitor election 2024-09-11T08:20:50.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: mon.b calling monitor election 2024-09-11T08:20:50.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:20:50.574 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.574 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.574 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:50.574 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.574 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: mon.c calling monitor election 2024-09-11T08:20:50.574 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.575 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:50.575 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.575 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:50.575 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-11T08:20:50.575 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: monmap e3: 3 mons at {a=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0],b=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0],c=[v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: fsmap 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: osdmap e4: 0 total, 0 up, 0 in 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: mgrmap e13: y(active, since 47s) 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: overall HEALTH_OK 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.576 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:50.883 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-11T08:20:50.883 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-11T08:20:51.161 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:51 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:51 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/888917666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:20:51.177 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:51 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:51.389 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:51 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:51.390 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:51 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/888917666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:20:51.390 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:51 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:51.390 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:51 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:51.390 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:51 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/888917666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-11T08:20:51.390 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:51 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:20:52.238 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:20:52.023+0000 7f26cdf9d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.240 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.241 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:52.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:53.338 INFO:teuthology.orchestra.run.smithi110.stdout:# minimal ceph.conf for 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:20:53.338 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-09-11T08:20:53.338 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 2850e098-7016-11ef-bcea-c7b262605968 2024-09-11T08:20:53.338 INFO:teuthology.orchestra.run.smithi110.stdout: mon_host = [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-11T08:20:53.338 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:20:53.338 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: Reconfiguring daemon mon.a on smithi110 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/212837289' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:20:53.339 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: Reconfiguring daemon mon.a on smithi110 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:53.340 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:53 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/212837289' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: Reconfiguring mon.a (unknown last config time)... 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:53.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: Reconfiguring daemon mon.a on smithi110 2024-09-11T08:20:53.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:53.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:53 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/212837289' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:53.933 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-11T08:20:53.933 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:20:53.934 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.conf 2024-09-11T08:20:53.963 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:20:53.963 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:20:54.028 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:20:54.029 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2024-09-11T08:20:54.061 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:20:54.061 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:20:54.128 INFO:tasks.cephadm:Adding mgr.y on smithi110 2024-09-11T08:20:54.129 INFO:tasks.cephadm:Adding mgr.x on smithi139 2024-09-11T08:20:54.129 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply mgr '2;smithi110=y;smithi139=x' 2024-09-11T08:20:54.294 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:54 smithi110 ceph-mon[28998]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:54.294 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:54 smithi110 ceph-mon[39404]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:54.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:54 smithi139 ceph-mon[32601]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:54.406 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:20:55.568 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: Reconfiguring mon.c (monmap changed)... 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: Reconfiguring daemon mon.c on smithi110 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[39404]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:55.569 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:55.570 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: Reconfiguring mon.c (monmap changed)... 2024-09-11T08:20:55.570 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:55.570 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:55.570 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:55.570 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: Reconfiguring daemon mon.c on smithi110 2024-09-11T08:20:55.570 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:55 smithi110 ceph-mon[28998]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:55.604 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:55.604 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:55.604 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: Reconfiguring mon.c (monmap changed)... 2024-09-11T08:20:55.604 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:55.604 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:55.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:55.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: Reconfiguring daemon mon.c on smithi110 2024-09-11T08:20:55.605 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:55 smithi139 ceph-mon[32601]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:55.982 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled mgr update... 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: Reconfiguring mon.b (monmap changed)... 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: Reconfiguring daemon mon.b on smithi139 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi110=y;smithi139=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: Saving service mgr spec with placement smithi110=y;smithi139=x;count:2 2024-09-11T08:20:56.820 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: Reconfiguring mon.b (monmap changed)... 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: Reconfiguring daemon mon.b on smithi139 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi110=y;smithi139=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: Saving service mgr spec with placement smithi110=y;smithi139=x;count:2 2024-09-11T08:20:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: Reconfiguring mon.b (monmap changed)... 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: Reconfiguring daemon mon.b on smithi139 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi110=y;smithi139=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: Saving service mgr spec with placement smithi110=y;smithi139=x;count:2 2024-09-11T08:20:56.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:57.147 DEBUG:teuthology.orchestra.run.smithi139:mgr.x> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.x.service 2024-09-11T08:20:57.150 INFO:tasks.cephadm:Deploying OSDs... 2024-09-11T08:20:57.150 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:20:57.150 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T08:20:57.173 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T08:20:57.173 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-09-11T08:20:57.231 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-11 08:20:44.512678884 +0000 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-11 08:14:23.554971299 +0000 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-11 08:14:23.554971299 +0000 2024-09-11T08:20:57.232 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-11 08:14:23.554971299 +0000 2024-09-11T08:20:57.233 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T08:20:57.299 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-11T08:20:57.300 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-11T08:20:57.300 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000131436 s, 3.9 MB/s 2024-09-11T08:20:57.301 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T08:20:57.361 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-09-11T08:20:57.420 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T08:20:57.420 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:57.420 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-11T08:20:57.420 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:57.420 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:57.421 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-11 08:20:44.513678868 +0000 2024-09-11T08:20:57.421 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-11 08:14:23.555971283 +0000 2024-09-11T08:20:57.421 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-11 08:14:23.555971283 +0000 2024-09-11T08:20:57.421 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-11 08:14:23.555971283 +0000 2024-09-11T08:20:57.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T08:20:57.487 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-11T08:20:57.488 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-11T08:20:57.488 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000153397 s, 3.3 MB/s 2024-09-11T08:20:57.489 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T08:20:57.549 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 963 Links: 1 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-11 08:20:44.513678868 +0000 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-11 08:14:23.554971299 +0000 2024-09-11T08:20:57.607 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-11 08:14:23.554971299 +0000 2024-09-11T08:20:57.608 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-11 08:14:23.554971299 +0000 2024-09-11T08:20:57.608 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T08:20:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-11T08:20:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-11T08:20:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000121772 s, 4.2 MB/s 2024-09-11T08:20:57.674 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T08:20:57.731 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-11 08:20:44.513678868 +0000 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-11 08:14:23.555971283 +0000 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-11 08:14:23.555971283 +0000 2024-09-11T08:20:57.788 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-11 08:14:23.555971283 +0000 2024-09-11T08:20:57.789 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T08:20:57.853 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-11T08:20:57.853 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-11T08:20:57.853 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000117549 s, 4.4 MB/s 2024-09-11T08:20:57.854 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T08:20:57.911 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:20:57.911 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T08:20:57.927 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T08:20:57.927 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2024-09-11T08:20:57.983 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-11 08:20:35.792226330 +0000 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-11 08:14:23.633010570 +0000 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-11 08:14:23.633010570 +0000 2024-09-11T08:20:57.984 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-11 08:14:23.633010570 +0000 2024-09-11T08:20:57.984 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T08:20:58.048 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-11T08:20:58.049 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-11T08:20:58.049 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000191202 s, 2.7 MB/s 2024-09-11T08:20:58.050 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T08:20:58.106 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2024-09-11T08:20:58.160 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 980 Links: 1 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-11 08:20:35.792226330 +0000 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-11 08:14:23.634010548 +0000 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-11 08:14:23.634010548 +0000 2024-09-11T08:20:58.161 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-11 08:14:23.634010548 +0000 2024-09-11T08:20:58.161 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T08:20:58.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:58.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:20:58.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:58.225 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-11T08:20:58.225 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-11T08:20:58.225 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000172053 s, 3.0 MB/s 2024-09-11T08:20:58.226 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T08:20:58.283 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2024-09-11T08:20:58.338 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T08:20:58.338 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:58.338 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-11T08:20:58.338 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:58.339 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:58.339 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-11 08:20:35.792226330 +0000 2024-09-11T08:20:58.339 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-11 08:14:23.634010548 +0000 2024-09-11T08:20:58.339 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-11 08:14:23.634010548 +0000 2024-09-11T08:20:58.339 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-11 08:14:23.634010548 +0000 2024-09-11T08:20:58.339 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T08:20:58.401 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-11T08:20:58.401 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-11T08:20:58.401 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000215733 s, 2.4 MB/s 2024-09-11T08:20:58.402 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T08:20:58.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:58.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:58.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:20:58.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:58.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:20:58.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:20:58.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:20:58.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-11T08:20:58.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:20:58.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:20:58.459 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2024-09-11T08:20:58.515 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T08:20:58.515 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:20:58.515 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-11T08:20:58.515 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:20:58.516 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T08:20:58.516 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2024-09-11 08:20:35.792226330 +0000 2024-09-11T08:20:58.516 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2024-09-11 08:14:23.633010570 +0000 2024-09-11T08:20:58.516 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2024-09-11 08:14:23.633010570 +0000 2024-09-11T08:20:58.516 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: 2024-09-11 08:14:23.633010570 +0000 2024-09-11T08:20:58.516 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T08:20:58.580 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2024-09-11T08:20:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2024-09-11T08:20:58.581 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000188409 s, 2.7 MB/s 2024-09-11T08:20:58.581 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T08:20:58.638 INFO:tasks.cephadm:Deploying osd.0 on smithi110 with /dev/vg_nvme/lv_4... 2024-09-11T08:20:58.638 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-11T08:20:58.884 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:20:59.271 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:58 smithi110 ceph-mon[28998]: Deploying daemon mgr.x on smithi139 2024-09-11T08:20:59.271 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:58 smithi110 ceph-mon[39404]: Deploying daemon mgr.x on smithi139 2024-09-11T08:20:59.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:58 smithi139 ceph-mon[32601]: Deploying daemon mgr.x on smithi139 2024-09-11T08:20:59.322 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 systemd[1]: Starting Ceph mgr.x for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:20:59.616 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 podman[34418]: 2024-09-11 08:20:59.259123941 +0000 UTC m=+0.015616597 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:20:59.616 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 podman[34418]: 2024-09-11 08:20:59.364579973 +0000 UTC m=+0.121072658 container create c0dfedff762315a0e4c76a77f513c31b2341b2e8733d0a8b8d1dcb1cd3c7f15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:20:59.616 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 podman[34418]: 2024-09-11 08:20:59.59174844 +0000 UTC m=+0.348241083 container init c0dfedff762315a0e4c76a77f513c31b2341b2e8733d0a8b8d1dcb1cd3c7f15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-11T08:20:59.617 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 podman[34418]: 2024-09-11 08:20:59.594921663 +0000 UTC m=+0.351414307 container start c0dfedff762315a0e4c76a77f513c31b2341b2e8733d0a8b8d1dcb1cd3c7f15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-11T08:20:59.617 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-mgr[34437]: -- 172.21.15.139:0/1521163337 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558091c62340 con 0x558091c6f000 2024-09-11T08:20:59.966 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 bash[34418]: c0dfedff762315a0e4c76a77f513c31b2341b2e8733d0a8b8d1dcb1cd3c7f15f 2024-09-11T08:20:59.967 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 systemd[1]: Started Ceph mgr.x for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:20:59.967 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:20:59.707+0000 7f7cc6af9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:20:59.967 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:20:59.766+0000 7f7cc6af9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:20:59.967 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:20:59.834+0000 7f7cc6af9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:20:59.967 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:20:59.965+0000 7f7cc6af9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:21:00.103 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[28998]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[39404]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.104 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:20:59 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:00.322 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:00.115+0000 7f7cc6af9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:21:00.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-mon[32601]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:00.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:00.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:20:59 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:00.467 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-11T08:21:00.548 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-11T08:21:00.559 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-09-11T08:21:00.559 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-09-11T08:21:00.559 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00861617 s, 1.2 GB/s 2024-09-11T08:21:00.559 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-09-11T08:21:00.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:00.391+0000 7f7cc6af9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:21:00.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:00.454+0000 7f7cc6af9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:21:00.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:00.619+0000 7f7cc6af9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:21:01.158 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_4 2024-09-11T08:21:01.807 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:01.476+0000 7f7cc6af9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:21:01.808 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:01.539+0000 7f7cc6af9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T08:21:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[28998]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:01.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T08:21:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:21:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:21:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:01 smithi110 ceph-mon[39404]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: Reconfiguring mgr.y (unknown last config time)... 2024-09-11T08:21:02.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:21:02.073 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:21:02.073 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:02.073 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:01 smithi139 ceph-mon[32601]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:21:02.181 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:02.535 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.248+0000 7f7cc6af9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:21:02.535 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.357+0000 7f7cc6af9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:21:02.535 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.475+0000 7f7cc6af9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:21:02.787 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.533+0000 7f7cc6af9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:21:02.787 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.657+0000 7f7cc6af9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:21:02.787 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.725+0000 7f7cc6af9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:21:02.787 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.785+0000 7f7cc6af9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:21:03.044 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.917+0000 7f7cc6af9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:21:03.044 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:02 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:02.979+0000 7f7cc6af9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:21:03.045 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:03.043+0000 7f7cc6af9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:21:03.296 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:03.122+0000 7f7cc6af9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:21:03.296 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:03.180+0000 7f7cc6af9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:21:03.296 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:03.236+0000 7f7cc6af9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:21:03.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:03.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:03.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:03.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.297 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.311 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.311 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:03.312 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:03.313 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:03.313 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:21:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:21:03.976+0000 7f7cc6af9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: Standby manager daemon x started 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:21:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:04 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:21:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: Standby manager daemon x started 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:04.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: Standby manager daemon x started 2024-09-11T08:21:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:21:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:21:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:21:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:04 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/3836871247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:21:05.391 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:05 smithi110 ceph-mon[39404]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:05.391 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:05 smithi110 ceph-mon[39404]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-11T08:21:05.391 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:05 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:21:05.391 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:05 smithi110 ceph-mon[28998]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:05.392 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:05 smithi110 ceph-mon[28998]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-11T08:21:05.392 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:05 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:21:05.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:05 smithi139 ceph-mon[32601]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:05.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:05 smithi139 ceph-mon[32601]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-11T08:21:05.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:05 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:21:06.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[28998]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:06.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/122724211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]: dispatch 2024-09-11T08:21:06.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]: dispatch 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]': finished 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[28998]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[39404]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/122724211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]: dispatch 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]: dispatch 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]': finished 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[39404]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T08:21:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:06 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:06 smithi139 ceph-mon[32601]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:06 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/122724211' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]: dispatch 2024-09-11T08:21:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:06 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]: dispatch 2024-09-11T08:21:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:06 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76330d63-fa1a-4768-9454-1d76c578f4cc"}]': finished 2024-09-11T08:21:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:06 smithi139 ceph-mon[32601]: osdmap e5: 1 total, 0 up, 1 in 2024-09-11T08:21:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:06 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:07.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:07 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3192204893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:07.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:07 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3192204893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:07.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:07 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3192204893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:08.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:08 smithi110 ceph-mon[28998]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:08.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:08 smithi110 ceph-mon[39404]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:08.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:08 smithi139 ceph-mon[32601]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:09.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:09 smithi110 ceph-mon[28998]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:09.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:09 smithi110 ceph-mon[39404]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:09.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:09 smithi139 ceph-mon[32601]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:11.727 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:11 smithi110 ceph-mon[28998]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:11.728 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:11 smithi110 ceph-mon[39404]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:11.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:11 smithi139 ceph-mon[32601]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:12.816 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:12 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-11T08:21:12.816 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:12 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:12.816 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:12 smithi110 ceph-mon[39404]: Deploying daemon osd.0 on smithi110 2024-09-11T08:21:12.816 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:12 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-11T08:21:12.816 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:12 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:12.816 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:12 smithi110 ceph-mon[28998]: Deploying daemon osd.0 on smithi110 2024-09-11T08:21:12.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:12 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-11T08:21:12.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:12 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:12.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:12 smithi139 ceph-mon[32601]: Deploying daemon osd.0 on smithi110 2024-09-11T08:21:13.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:13 smithi139 ceph-mon[32601]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:13.864 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:13 smithi110 ceph-mon[28998]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:13.864 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:13 smithi110 ceph-mon[39404]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:15.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:15 smithi139 ceph-mon[32601]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:15.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:15 smithi110 ceph-mon[28998]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:15.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:15 smithi110 ceph-mon[39404]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:16.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:16 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:16.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:16 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:16.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:16 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:16.823 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:16 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:16.823 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:16 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:16.823 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:16 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:16.823 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:16 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:16.824 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:16 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:16.824 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:16 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:17.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:17 smithi139 ceph-mon[32601]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:17.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:17 smithi110 ceph-mon[28998]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:17.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:17 smithi110 ceph-mon[39404]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:19.545 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.545 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.545 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[39404]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:19.545 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:19.545 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:19.545 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.546 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.546 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.546 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[28998]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:19.546 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:19.546 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:19.546 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.579 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 0 on host 'smithi110' 2024-09-11T08:21:19.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:19.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:19 smithi139 ceph-mon[32601]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:19.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:19.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:19.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:20.151 DEBUG:teuthology.orchestra.run.smithi110:osd.0> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.0.service 2024-09-11T08:21:20.154 INFO:tasks.cephadm:Deploying osd.1 on smithi110 with /dev/vg_nvme/lv_3... 2024-09-11T08:21:20.154 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-11T08:21:20.568 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[28998]: from='osd.0 v2:172.21.15.110:6801/1174545642' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:21:20.568 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[28998]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:21:20.568 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:20.568 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:20.568 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:20.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[39404]: from='osd.0 v2:172.21.15.110:6801/1174545642' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:21:20.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[39404]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:21:20.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:20.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:20.569 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:20 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:20.723 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:20 smithi139 ceph-mon[32601]: from='osd.0 v2:172.21.15.110:6801/1174545642' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:21:20.723 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:20 smithi139 ceph-mon[32601]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-11T08:21:20.723 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:20 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:20.723 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:20 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:20.723 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:20 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:21.401 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:21.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[28998]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[28998]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[28998]: from='osd.0 v2:172.21.15.110:6801/1174545642' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[28998]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[28998]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[39404]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[39404]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[39404]: from='osd.0 v2:172.21.15.110:6801/1174545642' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:21.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[39404]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:21.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-mon[39404]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:21.655 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0[43237]: 2024-09-11T08:21:21.392+0000 7f65e1b32640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T08:21:21.655 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:21:21 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0[43237]: 2024-09-11T08:21:21.395+0000 7f65d9941640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:21:21.741 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:21 smithi139 ceph-mon[32601]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-11T08:21:21.741 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:21 smithi139 ceph-mon[32601]: osdmap e6: 1 total, 0 up, 1 in 2024-09-11T08:21:21.741 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:21 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:21.741 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:21 smithi139 ceph-mon[32601]: from='osd.0 v2:172.21.15.110:6801/1174545642' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:21.742 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:21 smithi139 ceph-mon[32601]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:21.742 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:21 smithi139 ceph-mon[32601]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-11T08:21:22.515 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[28998]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:21:22.516 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[28998]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T08:21:22.516 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:22.516 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:22.516 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[39404]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:21:22.516 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[39404]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T08:21:22.516 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:22.516 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:22 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:22 smithi139 ceph-mon[32601]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:21:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:22 smithi139 ceph-mon[32601]: osdmap e7: 1 total, 0 up, 1 in 2024-09-11T08:21:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:22 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:22 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:23.260 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-11T08:21:23.364 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-11T08:21:23.376 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-09-11T08:21:23.376 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-09-11T08:21:23.376 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00953063 s, 1.1 GB/s 2024-09-11T08:21:23.376 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-09-11T08:21:23.575 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:21:23.575 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[28998]: osd.0 v2:172.21.15.110:6801/1174545642 boot 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[28998]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[28998]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[39404]: osd.0 v2:172.21.15.110:6801/1174545642 boot 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[39404]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:23.576 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:23 smithi110 ceph-mon[39404]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:23.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:23 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:21:23.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:23 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:21:23.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:23 smithi139 ceph-mon[32601]: osd.0 v2:172.21.15.110:6801/1174545642 boot 2024-09-11T08:21:23.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:23 smithi139 ceph-mon[32601]: osdmap e8: 1 total, 1 up, 1 in 2024-09-11T08:21:23.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:23 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:21:23.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:23 smithi139 ceph-mon[32601]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:24.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T08:21:24.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:24.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:24.551 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_3 2024-09-11T08:21:24.813 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: osdmap e9: 1 total, 1 up, 1 in 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:25.606 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:25 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi110 2024-09-11T08:21:25.606 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:25 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi110 to 16064M 2024-09-11T08:21:25.607 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:25 smithi110 ceph-mon[28998]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:25.607 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:25 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi110 2024-09-11T08:21:25.607 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:25 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi110 to 16064M 2024-09-11T08:21:25.607 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:25 smithi110 ceph-mon[39404]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:25.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:25 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi110 2024-09-11T08:21:25.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:25 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi110 to 16064M 2024-09-11T08:21:25.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:25 smithi139 ceph-mon[32601]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:26.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:26.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:26.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:26.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:27.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:27.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:27.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:27.875 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:27 smithi110 ceph-mon[28998]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:27.875 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:27 smithi110 ceph-mon[28998]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:27.875 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:27 smithi110 ceph-mon[39404]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:27.875 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:27 smithi110 ceph-mon[39404]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:28.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:27 smithi139 ceph-mon[32601]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:28.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:27 smithi139 ceph-mon[32601]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:28.853 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1652366329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]: dispatch 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]: dispatch 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]': finished 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[28998]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1652366329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]: dispatch 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]: dispatch 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]': finished 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[39404]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T08:21:28.854 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:28 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:29.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:28 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1652366329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]: dispatch 2024-09-11T08:21:29.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:28 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]: dispatch 2024-09-11T08:21:29.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:28 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31c2d1f0-0aa3-415c-8fc9-8e64ae860768"}]': finished 2024-09-11T08:21:29.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:28 smithi139 ceph-mon[32601]: osdmap e10: 2 total, 1 up, 2 in 2024-09-11T08:21:29.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:28 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:29 smithi110 ceph-mon[28998]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:29 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3514974219' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:29.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:29 smithi110 ceph-mon[39404]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:29.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:29 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3514974219' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:30.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:29 smithi139 ceph-mon[32601]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:30.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:29 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3514974219' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:31.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:31 smithi110 ceph-mon[28998]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:31.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:31 smithi110 ceph-mon[39404]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:32.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:31 smithi139 ceph-mon[32601]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:33 smithi110 ceph-mon[28998]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:33.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:33 smithi110 ceph-mon[39404]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:34.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:33 smithi139 ceph-mon[32601]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:34.873 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:34 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-11T08:21:34.873 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:34 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:34.873 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:34 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-11T08:21:34.873 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:34 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:35.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:34 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-11T08:21:35.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:34 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:35 smithi139 ceph-mon[32601]: Deploying daemon osd.1 on smithi110 2024-09-11T08:21:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:35 smithi139 ceph-mon[32601]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:35.855 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:35 smithi110 ceph-mon[28998]: Deploying daemon osd.1 on smithi110 2024-09-11T08:21:35.855 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:35 smithi110 ceph-mon[28998]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:35.856 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:35 smithi110 ceph-mon[39404]: Deploying daemon osd.1 on smithi110 2024-09-11T08:21:35.856 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:35 smithi110 ceph-mon[39404]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:37.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:37 smithi110 ceph-mon[39404]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:37.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:37 smithi110 ceph-mon[28998]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:38.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:37 smithi139 ceph-mon[32601]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:39.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[28998]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:39.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:39.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:39.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:39.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[39404]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:39.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:39.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:39.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:39 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:40.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:39 smithi139 ceph-mon[32601]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:40.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:39 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:40.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:39 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:40.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:39 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[28998]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[39404]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:41.943 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 1 on host 'smithi110' 2024-09-11T08:21:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:41 smithi139 ceph-mon[32601]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:42.511 DEBUG:teuthology.orchestra.run.smithi110:osd.1> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.1.service 2024-09-11T08:21:42.513 INFO:tasks.cephadm:Deploying osd.2 on smithi110 with /dev/vg_nvme/lv_2... 2024-09-11T08:21:42.513 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-11T08:21:42.837 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:42.838 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:42.838 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:42.838 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[39404]: from='osd.1 v2:172.21.15.110:6805/1566634575' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:21:42.838 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[39404]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:21:42.838 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:42.838 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:42.839 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:42.839 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[28998]: from='osd.1 v2:172.21.15.110:6805/1566634575' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:21:42.839 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:42 smithi110 ceph-mon[28998]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:21:43.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:21:43.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:43.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:43.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:42 smithi139 ceph-mon[32601]: from='osd.1 v2:172.21.15.110:6805/1566634575' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:21:43.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:42 smithi139 ceph-mon[32601]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-11T08:21:44.011 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:44.132 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[28998]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:44.132 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[28998]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T08:21:44.132 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[28998]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T08:21:44.132 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:44.132 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[28998]: from='osd.1 v2:172.21.15.110:6805/1566634575' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:44.132 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[28998]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:44.133 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[39404]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:44.133 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[39404]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T08:21:44.133 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[39404]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T08:21:44.133 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:44.133 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[39404]: from='osd.1 v2:172.21.15.110:6805/1566634575' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:44.133 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-mon[39404]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:44.133 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1[48826]: 2024-09-11T08:21:43.953+0000 7fa11a335640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T08:21:44.133 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:21:43 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1[48826]: 2024-09-11T08:21:43.956+0000 7fa11496b640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:21:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:43 smithi139 ceph-mon[32601]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:43 smithi139 ceph-mon[32601]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-11T08:21:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:43 smithi139 ceph-mon[32601]: osdmap e11: 2 total, 1 up, 2 in 2024-09-11T08:21:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:43 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:43 smithi139 ceph-mon[32601]: from='osd.1 v2:172.21.15.110:6805/1566634575' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:43 smithi139 ceph-mon[32601]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:21:45.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[28998]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:21:45.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[28998]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T08:21:45.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:45.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:45.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[39404]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:21:45.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[39404]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T08:21:45.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:45.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:44 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:45.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:44 smithi139 ceph-mon[32601]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:21:45.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:44 smithi139 ceph-mon[32601]: osdmap e12: 2 total, 1 up, 2 in 2024-09-11T08:21:45.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:44 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:45.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:44 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:46.149 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-11T08:21:46.232 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-11T08:21:46.232 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:21:46.232 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:21:46.232 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:46.232 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:46.232 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: osd.1 v2:172.21.15.110:6805/1566634575 boot 2024-09-11T08:21:46.232 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T08:21:46.232 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:21:46.233 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: osd.1 v2:172.21.15.110:6805/1566634575 boot 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:46.234 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:45 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.244 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-09-11T08:21:46.244 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-09-11T08:21:46.244 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010622 s, 987 MB/s 2024-09-11T08:21:46.244 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: osd.1 v2:172.21.15.110:6805/1566634575 boot 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: osdmap e13: 2 total, 2 up, 2 in 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:21:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:46.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:21:46.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:46.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:21:46.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:45 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:21:46.763 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_2 2024-09-11T08:21:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:21:47.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:46 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi110 2024-09-11T08:21:47.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:46 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi110 to 8032M 2024-09-11T08:21:47.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:46 smithi110 ceph-mon[39404]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T08:21:47.123 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:46 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi110 2024-09-11T08:21:47.123 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:46 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi110 to 8032M 2024-09-11T08:21:47.123 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:46 smithi110 ceph-mon[28998]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T08:21:47.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:46 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi110 2024-09-11T08:21:47.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:46 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi110 to 8032M 2024-09-11T08:21:47.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:46 smithi139 ceph-mon[32601]: osdmap e14: 2 total, 2 up, 2 in 2024-09-11T08:21:48.080 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:47 smithi110 ceph-mon[28998]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:48.080 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:47 smithi110 ceph-mon[39404]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:48.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:47 smithi139 ceph-mon[32601]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:49.286 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:48 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:49.286 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:48 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:49.286 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:48 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:49.286 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:48 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:49.286 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:48 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:49.286 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:48 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:49.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:48 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:21:49.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:48 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:21:49.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:48 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:50.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:49 smithi139 ceph-mon[32601]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:50.223 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:49 smithi139 ceph-mon[32601]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:50.228 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:49 smithi110 ceph-mon[28998]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:50.228 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:49 smithi110 ceph-mon[28998]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:50.228 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:49 smithi110 ceph-mon[39404]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:50.229 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:49 smithi110 ceph-mon[39404]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:21:51.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:50 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3731367948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cdaba0-6b9f-4716-9e16-dd708fe2cc74"}]: dispatch 2024-09-11T08:21:51.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:50 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3731367948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48cdaba0-6b9f-4716-9e16-dd708fe2cc74"}]': finished 2024-09-11T08:21:51.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:50 smithi139 ceph-mon[32601]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T08:21:51.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:21:51.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3731367948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cdaba0-6b9f-4716-9e16-dd708fe2cc74"}]: dispatch 2024-09-11T08:21:51.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3731367948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48cdaba0-6b9f-4716-9e16-dd708fe2cc74"}]': finished 2024-09-11T08:21:51.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[28998]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T08:21:51.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:21:51.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3731367948' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48cdaba0-6b9f-4716-9e16-dd708fe2cc74"}]: dispatch 2024-09-11T08:21:51.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3731367948' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48cdaba0-6b9f-4716-9e16-dd708fe2cc74"}]': finished 2024-09-11T08:21:51.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[39404]: osdmap e15: 3 total, 2 up, 3 in 2024-09-11T08:21:51.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:50 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:21:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:51 smithi139 ceph-mon[32601]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:51 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2861905650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:51 smithi110 ceph-mon[28998]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:52.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:51 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2861905650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:51 smithi110 ceph-mon[39404]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:51 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2861905650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:21:54.248 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:53 smithi110 ceph-mon[28998]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:54.248 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:53 smithi110 ceph-mon[39404]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:54.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:53 smithi139 ceph-mon[32601]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:56.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:55 smithi110 ceph-mon[39404]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:56.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:55 smithi110 ceph-mon[28998]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:56.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:55 smithi139 ceph-mon[32601]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:57.230 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-11T08:21:57.231 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:56 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:57.231 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-11T08:21:57.231 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:56 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:57.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-11T08:21:57.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:56 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:21:58.134 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:57 smithi110 ceph-mon[28998]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:58.135 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:57 smithi110 ceph-mon[28998]: Deploying daemon osd.2 on smithi110 2024-09-11T08:21:58.135 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:57 smithi110 ceph-mon[39404]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:58.135 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:57 smithi110 ceph-mon[39404]: Deploying daemon osd.2 on smithi110 2024-09-11T08:21:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:57 smithi139 ceph-mon[32601]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:21:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:57 smithi139 ceph-mon[32601]: Deploying daemon osd.2 on smithi110 2024-09-11T08:22:00.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:21:59 smithi139 ceph-mon[32601]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:21:59 smithi110 ceph-mon[28998]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:00.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:21:59 smithi110 ceph-mon[39404]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:01 smithi139 ceph-mon[32601]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:02.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[39404]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:02.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:02.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:02.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:02.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[28998]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:02.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:02.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:02.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.068 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[39404]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:04.068 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.068 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.068 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:04.068 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:04.068 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.068 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[28998]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:04.069 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.069 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.069 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:04.069 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:04.069 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:03 smithi139 ceph-mon[32601]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:04.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:04.367 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 2 on host 'smithi110' 2024-09-11T08:22:04.971 DEBUG:teuthology.orchestra.run.smithi110:osd.2> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.2.service 2024-09-11T08:22:04.973 INFO:tasks.cephadm:Deploying osd.3 on smithi110 with /dev/vg_nvme/lv_1... 2024-09-11T08:22:04.974 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-11T08:22:05.198 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[39404]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-11T08:22:05.199 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:05.199 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:05.199 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:05.199 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[28998]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-11T08:22:05.199 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:05.199 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:05.199 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:05.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:04 smithi139 ceph-mon[32601]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-11T08:22:05.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:05.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:05.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:06.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[28998]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:06.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[28998]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T08:22:06.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[28998]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T08:22:06.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:06.281 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[28998]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:22:06.282 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[39404]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:06.282 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[39404]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T08:22:06.282 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[39404]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T08:22:06.282 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:06.282 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-mon[39404]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:22:06.282 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2[54806]: 2024-09-11T08:22:05.995+0000 7f5c27e2c640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T08:22:06.282 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2[54806]: 2024-09-11T08:22:05.998+0000 7f5c24c67640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:22:06.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:05 smithi139 ceph-mon[32601]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:06.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:05 smithi139 ceph-mon[32601]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-11T08:22:06.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:05 smithi139 ceph-mon[32601]: osdmap e16: 3 total, 2 up, 3 in 2024-09-11T08:22:06.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:05 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:06.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:05 smithi139 ceph-mon[32601]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:22:06.414 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:07.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[28998]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:22:07.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[28998]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T08:22:07.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[39404]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:22:07.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[39404]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T08:22:07.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:07 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:07 smithi139 ceph-mon[32601]: from='osd.2 v2:172.21.15.110:6809/2039773886' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:22:07.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:07 smithi139 ceph-mon[32601]: osdmap e17: 3 total, 2 up, 3 in 2024-09-11T08:22:07.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:07 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:07 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:07.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:07 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:08.294 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: osd.2 v2:172.21.15.110:6809/2039773886 boot 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.295 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: osd.2 v2:172.21.15.110:6809/2039773886 boot 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.296 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.297 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.297 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.297 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.297 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:08.297 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:08.297 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:08 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: osd.2 v2:172.21.15.110:6809/2039773886 boot 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: osdmap e18: 3 total, 3 up, 3 in 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:08.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:08.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:08.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:08 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-11T08:22:08.701 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-11T08:22:08.713 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-09-11T08:22:08.713 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-09-11T08:22:08.713 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00954525 s, 1.1 GB/s 2024-09-11T08:22:08.713 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-09-11T08:22:09.283 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi110 2024-09-11T08:22:09.284 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi110 to 5354M 2024-09-11T08:22:09.284 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[28998]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T08:22:09.284 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-11T08:22:09.284 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi110 2024-09-11T08:22:09.284 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi110 to 5354M 2024-09-11T08:22:09.284 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[39404]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T08:22:09.284 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:09 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-11T08:22:09.309 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_1 2024-09-11T08:22:09.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:09 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi110 2024-09-11T08:22:09.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:09 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi110 to 5354M 2024-09-11T08:22:09.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:09 smithi139 ceph-mon[32601]: osdmap e19: 3 total, 3 up, 3 in 2024-09-11T08:22:09.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:09 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-11T08:22:09.551 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:22:10.265 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[39404]: pgmap v72: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:10.265 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-11T08:22:10.265 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[39404]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T08:22:10.265 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:22:10.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[28998]: pgmap v72: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:10.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-11T08:22:10.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[28998]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T08:22:10.266 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:10 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:22:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:10 smithi139 ceph-mon[32601]: pgmap v72: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:10 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-11T08:22:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:10 smithi139 ceph-mon[32601]: osdmap e20: 3 total, 3 up, 3 in 2024-09-11T08:22:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:10 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-11T08:22:10.927 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:10 smithi110 sudo[58984]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:22:10.927 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:10 smithi110 sudo[58984]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:10.927 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:10 smithi110 sudo[58984]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:10.928 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:10 smithi110 sudo[58984]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:11.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:11 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:22:11.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:11 smithi139 ceph-mon[32601]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T08:22:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:11 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:22:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:11 smithi110 ceph-mon[28998]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T08:22:11.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:11 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-11T08:22:11.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:11 smithi110 ceph-mon[39404]: osdmap e21: 3 total, 3 up, 3 in 2024-09-11T08:22:11.404 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59008]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:22:11.404 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59008]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:11.404 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59008]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:11.404 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59008]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:11.680 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59085]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:22:11.680 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59085]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:11.680 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59085]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:11.680 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59085]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:11.930 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:22:11.931 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59136]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:22:11.931 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59136]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:11.931 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59136]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:11.931 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59136]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:12.191 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:12.191 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59156]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:12.192 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:11 smithi110 sudo[59156]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:12.192 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[28998]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:12.192 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[28998]: osdmap e22: 3 total, 3 up, 3 in 2024-09-11T08:22:12.192 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:12.192 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:12.192 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:12.192 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:12.192 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[39404]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:12.193 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[39404]: osdmap e22: 3 total, 3 up, 3 in 2024-09-11T08:22:12.193 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:12.193 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:12.193 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:12.193 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:12 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:12.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 ceph-mon[32601]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:12.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 ceph-mon[32601]: osdmap e22: 3 total, 3 up, 3 in 2024-09-11T08:22:12.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:12.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:12.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:12.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:12.452 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59165]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:22:12.453 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59165]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:12.453 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59165]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:12.453 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59165]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:12.453 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59162]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:22:12.453 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59162]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:12.453 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59162]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:12.453 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59162]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:12.704 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59189]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:22:12.980 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59189]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:12.980 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59189]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:12.981 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:12 smithi110 sudo[59189]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:13.203 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 sudo[35072]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:22:13.204 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 sudo[35072]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:13.204 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 sudo[35072]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:13.204 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:12 smithi139 sudo[35072]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:13.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:13.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:13.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:13.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:13.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:13.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:13.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:13.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 sudo[35075]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:22:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 sudo[35075]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 sudo[35075]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 sudo[35075]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:13.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59240]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-11T08:22:13.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:13.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59240]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:13.738 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59240]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:14.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59243]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-11T08:22:14.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59243]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-11T08:22:14.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59243]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-11T08:22:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:13 smithi110 sudo[59243]: pam_unix(sudo:session): session closed for user root 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: pgmap v77: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:14.518 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2443429965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6b86009-9a77-4998-913a-63e9c9ff1cf2"}]: dispatch 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2443429965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6b86009-9a77-4998-913a-63e9c9ff1cf2"}]': finished 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: pgmap v77: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:14.519 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2443429965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6b86009-9a77-4998-913a-63e9c9ff1cf2"}]: dispatch 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2443429965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6b86009-9a77-4998-913a-63e9c9ff1cf2"}]': finished 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T08:22:14.520 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:14 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: pgmap v77: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:22:14.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:22:14.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:22:14.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2443429965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6b86009-9a77-4998-913a-63e9c9ff1cf2"}]: dispatch 2024-09-11T08:22:14.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-11T08:22:14.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2443429965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6b86009-9a77-4998-913a-63e9c9ff1cf2"}]': finished 2024-09-11T08:22:14.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: osdmap e23: 4 total, 3 up, 4 in 2024-09-11T08:22:14.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:14 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:15.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:15 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/479398804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:22:15.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:15 smithi139 ceph-mon[32601]: pgmap v79: 1 pgs: 1 active+clean; 320 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:15.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:15 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/479398804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:22:15.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:15 smithi110 ceph-mon[28998]: pgmap v79: 1 pgs: 1 active+clean; 320 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:15.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:15 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/479398804' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:22:15.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:15 smithi110 ceph-mon[39404]: pgmap v79: 1 pgs: 1 active+clean; 320 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:17.798 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:17 smithi110 ceph-mon[28998]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.9 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-09-11T08:22:17.798 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:17 smithi110 ceph-mon[39404]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.9 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-09-11T08:22:17.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:17 smithi139 ceph-mon[32601]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.9 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-09-11T08:22:19.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:19 smithi110 ceph-mon[28998]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.6 KiB/s rd, 174 KiB/s wr, 12 op/s 2024-09-11T08:22:19.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:19 smithi110 ceph-mon[39404]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.6 KiB/s rd, 174 KiB/s wr, 12 op/s 2024-09-11T08:22:20.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:19 smithi139 ceph-mon[32601]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.6 KiB/s rd, 174 KiB/s wr, 12 op/s 2024-09-11T08:22:20.808 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:20 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-11T08:22:20.809 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:20 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:20.809 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:20 smithi110 ceph-mon[39404]: Deploying daemon osd.3 on smithi110 2024-09-11T08:22:20.809 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:20 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-11T08:22:20.809 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:20 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:20.809 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:20 smithi110 ceph-mon[28998]: Deploying daemon osd.3 on smithi110 2024-09-11T08:22:20.973 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:20 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-11T08:22:20.973 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:20 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:20.973 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:20 smithi139 ceph-mon[32601]: Deploying daemon osd.3 on smithi110 2024-09-11T08:22:21.644 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:21 smithi110 ceph-mon[28998]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-09-11T08:22:21.644 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:21 smithi110 ceph-mon[39404]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-09-11T08:22:22.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:21 smithi139 ceph-mon[32601]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 137 KiB/s wr, 9 op/s 2024-09-11T08:22:23.883 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:23 smithi110 ceph-mon[39404]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-11T08:22:23.883 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:23 smithi110 ceph-mon[28998]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-11T08:22:24.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:23 smithi139 ceph-mon[32601]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-11T08:22:24.865 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:24.865 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:24.865 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:24 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:24.866 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:24.866 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:24.866 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:24 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:25.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:25.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:25.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:24 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:25.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:25 smithi110 ceph-mon[28998]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.5 KiB/s rd, 121 KiB/s wr, 8 op/s 2024-09-11T08:22:25.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:25 smithi110 ceph-mon[39404]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.5 KiB/s rd, 121 KiB/s wr, 8 op/s 2024-09-11T08:22:26.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:25 smithi139 ceph-mon[32601]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.5 KiB/s rd, 121 KiB/s wr, 8 op/s 2024-09-11T08:22:27.621 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 3 on host 'smithi110' 2024-09-11T08:22:27.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:27 smithi110 ceph-mon[39404]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-11T08:22:27.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:27 smithi110 ceph-mon[28998]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-11T08:22:28.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:27 smithi139 ceph-mon[32601]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-09-11T08:22:28.141 DEBUG:teuthology.orchestra.run.smithi110:osd.3> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.3.service 2024-09-11T08:22:28.144 INFO:tasks.cephadm:Deploying osd.4 on smithi139 with /dev/vg_nvme/lv_4... 2024-09-11T08:22:28.144 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-11T08:22:28.394 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:22:28.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:28 smithi139 ceph-mon[32601]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:22:28.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:28 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:28 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:28 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.806 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:28 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[28998]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:22:28.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[39404]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-11T08:22:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:28.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:28 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:29.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[28998]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:29.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[28998]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T08:22:29.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[28998]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T08:22:29.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:29.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[28998]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:22:29.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[39404]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:29.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[39404]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T08:22:29.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[39404]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T08:22:29.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:29.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-mon[39404]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:22:29.655 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3[60913]: 2024-09-11T08:22:29.595+0000 7f2d9a43b640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T08:22:29.655 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:22:29 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3[60913]: 2024-09-11T08:22:29.599+0000 7f2d94a71640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:22:29.708 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:29 smithi139 ceph-mon[32601]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:29.708 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:29 smithi139 ceph-mon[32601]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-11T08:22:29.708 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:29 smithi139 ceph-mon[32601]: osdmap e24: 4 total, 3 up, 4 in 2024-09-11T08:22:29.708 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:29 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:29.708 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:29 smithi139 ceph-mon[32601]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-09-11T08:22:30.075 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-11T08:22:30.190 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-11T08:22:30.203 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-09-11T08:22:30.203 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-09-11T08:22:30.204 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114092 s, 919 MB/s 2024-09-11T08:22:30.204 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-09-11T08:22:30.797 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2024-09-11T08:22:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:22:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:22:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[28998]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:22:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[28998]: osdmap e25: 4 total, 3 up, 4 in 2024-09-11T08:22:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:22:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:22:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[39404]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:22:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[39404]: osdmap e25: 4 total, 3 up, 4 in 2024-09-11T08:22:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:30 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:31.036 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:22:31.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:30 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:22:31.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:30 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:22:31.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:30 smithi139 ceph-mon[32601]: from='osd.3 v2:172.21.15.110:6813/3504281989' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-09-11T08:22:31.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:30 smithi139 ceph-mon[32601]: osdmap e25: 4 total, 3 up, 4 in 2024-09-11T08:22:31.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:30 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:31.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:30 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:31.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:30 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:31.812 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:31.812 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: osd.3 v2:172.21.15.110:6813/3504281989 boot 2024-09-11T08:22:31.812 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:31.813 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:31.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: osd.3 v2:172.21.15.110:6813/3504281989 boot 2024-09-11T08:22:31.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T08:22:31.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-11T08:22:31.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: osd.3 v2:172.21.15.110:6813/3504281989 boot 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: osdmap e26: 4 total, 4 up, 4 in 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:31.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:32.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:32 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi110 2024-09-11T08:22:32.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:32 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi110 to 4016M 2024-09-11T08:22:32.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:32 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:32.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:32 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:32.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:32 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:32.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:32 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:32.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi110 2024-09-11T08:22:32.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi110 to 4016M 2024-09-11T08:22:32.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:32.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:32.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:32.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:32.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi110 2024-09-11T08:22:32.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi110 to 4016M 2024-09-11T08:22:32.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:32.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:32.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:32.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:32 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:33.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:33 smithi139 ceph-mon[32601]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:33.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:33 smithi139 ceph-mon[32601]: osdmap e27: 4 total, 4 up, 4 in 2024-09-11T08:22:33.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:33 smithi139 ceph-mon[32601]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:33.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:33 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:33.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:33 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:33.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:33 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[28998]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[28998]: osdmap e27: 4 total, 4 up, 4 in 2024-09-11T08:22:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[28998]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[39404]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[39404]: osdmap e27: 4 total, 4 up, 4 in 2024-09-11T08:22:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[39404]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:33 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:35.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:35 smithi139 ceph-mon[32601]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:35.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:35 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/925624285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]: dispatch 2024-09-11T08:22:35.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:35 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]: dispatch 2024-09-11T08:22:35.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:35 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]': finished 2024-09-11T08:22:35.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:35 smithi139 ceph-mon[32601]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T08:22:35.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:35 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:35.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:35 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/2160055697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:22:35.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[28998]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:35.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/925624285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]: dispatch 2024-09-11T08:22:35.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]: dispatch 2024-09-11T08:22:35.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]': finished 2024-09-11T08:22:35.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[28998]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/2160055697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[39404]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/925624285' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]: dispatch 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]: dispatch 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a513f216-1a9e-4fd2-9584-f28380c9e427"}]': finished 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[39404]: osdmap e28: 5 total, 4 up, 5 in 2024-09-11T08:22:35.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:35.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:35 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/2160055697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:22:37.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:37 smithi110 ceph-mon[28998]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:37.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:37 smithi110 ceph-mon[39404]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:38.038 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:37 smithi139 ceph-mon[32601]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:39.704 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:39 smithi139 ceph-mon[32601]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:39.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:39 smithi110 ceph-mon[28998]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:39.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:39 smithi110 ceph-mon[39404]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[28998]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-11T08:22:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:41.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[28998]: Deploying daemon osd.4 on smithi139 2024-09-11T08:22:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[39404]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-11T08:22:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:41.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:41 smithi110 ceph-mon[39404]: Deploying daemon osd.4 on smithi139 2024-09-11T08:22:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:41 smithi139 ceph-mon[32601]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-11T08:22:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:42.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:41 smithi139 ceph-mon[32601]: Deploying daemon osd.4 on smithi139 2024-09-11T08:22:43.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:43 smithi110 ceph-mon[28998]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:43.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:43 smithi110 ceph-mon[39404]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:43.999 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:43 smithi139 ceph-mon[32601]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:45.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:45 smithi139 ceph-mon[32601]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:45.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:45 smithi110 ceph-mon[28998]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:45.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:45 smithi110 ceph-mon[39404]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:46.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:46 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:46.805 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:46 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:46.806 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:46 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:46.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:46 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:46.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:46 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:46.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:46 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:46.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:46 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:46.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:46 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:46.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:46 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:47.784 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:47 smithi139 ceph-mon[32601]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:47.784 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:47 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:47.784 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:47 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:48.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:47 smithi110 ceph-mon[28998]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:48.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:47 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:48.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:47 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:48.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:47 smithi110 ceph-mon[39404]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:48.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:47 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:48.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:47 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:48.782 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:48 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:48.782 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:48 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:48.782 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:48 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:49.101 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 4 on host 'smithi139' 2024-09-11T08:22:49.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:48 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:49.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:48 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:49.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:48 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:49.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:48 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:49.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:48 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:49.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:48 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:49.708 DEBUG:teuthology.orchestra.run.smithi139:osd.4> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.4.service 2024-09-11T08:22:49.710 INFO:tasks.cephadm:Deploying osd.5 on smithi139 with /dev/vg_nvme/lv_3... 2024-09-11T08:22:49.710 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-11T08:22:50.020 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:49 smithi139 ceph-mon[32601]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:50.020 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:49 smithi139 ceph-mon[32601]: from='osd.4 v2:172.21.15.139:6800/2716089259' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:22:50.020 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:49 smithi139 ceph-mon[32601]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:22:50.021 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:49 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:50.021 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:49 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:50.021 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:49 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:50.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[28998]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:50.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[28998]: from='osd.4 v2:172.21.15.139:6800/2716089259' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:22:50.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[28998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:22:50.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[39404]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[39404]: from='osd.4 v2:172.21.15.139:6800/2716089259' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[39404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:50.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:49 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:50.919 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:50 smithi139 ceph-mon[32601]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T08:22:50.919 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:50 smithi139 ceph-mon[32601]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T08:22:50.919 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:50.919 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:50 smithi139 ceph-mon[32601]: from='osd.4 v2:172.21.15.139:6800/2716089259' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:22:50.919 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:50 smithi139 ceph-mon[32601]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:22:50.919 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:22:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:22:50.693+0000 7f52338a9640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T08:22:50.920 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:22:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:22:50.698+0000 7f522f6e2640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:22:50.933 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:22:51.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[28998]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T08:22:51.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[28998]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[28998]: from='osd.4 v2:172.21.15.139:6800/2716089259' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[28998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[39404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[39404]: osdmap e29: 5 total, 4 up, 5 in 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[39404]: from='osd.4 v2:172.21.15.139:6800/2716089259' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:22:51.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:50 smithi110 ceph-mon[39404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:22:51.880 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:51.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:22:51.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T08:22:51.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:51.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:51.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: osd.4 v2:172.21.15.139:6800/2716089259 boot 2024-09-11T08:22:51.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T08:22:51.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:51 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:52.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:52.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:22:52.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: osd.4 v2:172.21.15.139:6800/2716089259 boot 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: osdmap e30: 5 total, 4 up, 5 in 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:52.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: osd.4 v2:172.21.15.139:6800/2716089259 boot 2024-09-11T08:22:52.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: osdmap e31: 5 total, 5 up, 5 in 2024-09-11T08:22:52.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:51 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:22:52.840 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:52 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:22:52.841 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:52 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:22:52.841 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:52 smithi139 ceph-mon[32601]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T08:22:53.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:52 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:22:53.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:52 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:22:53.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:52 smithi110 ceph-mon[28998]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T08:22:53.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:52 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:22:53.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:52 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:22:53.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:52 smithi110 ceph-mon[39404]: osdmap e32: 5 total, 5 up, 5 in 2024-09-11T08:22:53.186 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-11T08:22:53.277 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-11T08:22:53.292 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-09-11T08:22:53.292 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-09-11T08:22:53.292 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125905 s, 833 MB/s 2024-09-11T08:22:53.292 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-09-11T08:22:54.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:22:54.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T08:22:54.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:54.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:54 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: osdmap e33: 5 total, 5 up, 5 in 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:22:54.405 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.406 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:54.406 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:22:54.406 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:54 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:22:54.429 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2024-09-11T08:22:54.665 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:22:55.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:55 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi139 2024-09-11T08:22:55.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:55 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi139 to 17088M 2024-09-11T08:22:55.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:55 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi139 2024-09-11T08:22:55.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:55 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi139 to 17088M 2024-09-11T08:22:55.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:55 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi139 2024-09-11T08:22:55.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:55 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi139 to 17088M 2024-09-11T08:22:56.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:56 smithi139 ceph-mon[32601]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:22:56.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:56 smithi110 ceph-mon[28998]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:22:56.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:56 smithi110 ceph-mon[39404]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:22:57.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:57.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:57.324 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:57.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:57.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:57.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:57.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:22:57.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:22:57.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:22:58.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:58 smithi110 ceph-mon[39404]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:58.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:58 smithi110 ceph-mon[39404]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-11T08:22:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:58 smithi110 ceph-mon[28998]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:58.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:58 smithi110 ceph-mon[28998]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-11T08:22:58.438 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:58 smithi139 ceph-mon[32601]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:22:58.438 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:58 smithi139 ceph-mon[32601]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-11T08:22:59.277 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:59 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/938851237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]: dispatch 2024-09-11T08:22:59.277 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:59 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]: dispatch 2024-09-11T08:22:59.277 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:59 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]': finished 2024-09-11T08:22:59.277 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:59 smithi139 ceph-mon[32601]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T08:22:59.277 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:22:59 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:22:59.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/938851237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]: dispatch 2024-09-11T08:22:59.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]: dispatch 2024-09-11T08:22:59.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]': finished 2024-09-11T08:22:59.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[28998]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T08:22:59.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:22:59.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/938851237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]: dispatch 2024-09-11T08:22:59.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]: dispatch 2024-09-11T08:22:59.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "543c8080-fb5e-4ebe-aea6-8932d7e49b96"}]': finished 2024-09-11T08:22:59.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[39404]: osdmap e34: 6 total, 5 up, 6 in 2024-09-11T08:22:59.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:22:59 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:00 smithi110 ceph-mon[28998]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-11T08:23:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:00 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/2716315059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:00.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:00 smithi110 ceph-mon[39404]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-11T08:23:00.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:00 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/2716315059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:00.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:00 smithi139 ceph-mon[32601]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-11T08:23:00.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:00 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/2716315059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:02 smithi139 ceph-mon[32601]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:23:02.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:02 smithi110 ceph-mon[28998]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:23:02.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:02 smithi110 ceph-mon[39404]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-11T08:23:04.338 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:04 smithi139 ceph-mon[32601]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-11T08:23:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:04 smithi110 ceph-mon[28998]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-11T08:23:04.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:04 smithi110 ceph-mon[39404]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-11T08:23:05.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:05 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-11T08:23:05.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:05 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:05.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:05 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-11T08:23:05.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:05 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:05.461 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:05 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-11T08:23:05.461 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:05 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:06.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:06 smithi110 ceph-mon[28998]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:23:06.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:06 smithi110 ceph-mon[28998]: Deploying daemon osd.5 on smithi139 2024-09-11T08:23:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:06 smithi110 ceph-mon[39404]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:23:06.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:06 smithi110 ceph-mon[39404]: Deploying daemon osd.5 on smithi139 2024-09-11T08:23:06.420 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:06 smithi139 ceph-mon[32601]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:23:06.420 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:06 smithi139 ceph-mon[32601]: Deploying daemon osd.5 on smithi139 2024-09-11T08:23:08.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:08 smithi139 ceph-mon[32601]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:08.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:08 smithi110 ceph-mon[28998]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:08.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:08 smithi110 ceph-mon[39404]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:10 smithi139 ceph-mon[32601]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:10 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:10 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:10.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:10 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:10.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[28998]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:10.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:10.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:10.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:10.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[39404]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:10.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:10.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:10.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:10 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:12.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:12 smithi110 ceph-mon[28998]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:12.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:12 smithi110 ceph-mon[39404]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:12.415 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:12 smithi139 ceph-mon[32601]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:12.706 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 5 on host 'smithi139' 2024-09-11T08:23:13.351 DEBUG:teuthology.orchestra.run.smithi139:osd.5> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.5.service 2024-09-11T08:23:13.354 INFO:tasks.cephadm:Deploying osd.6 on smithi139 with /dev/vg_nvme/lv_2... 2024-09-11T08:23:13.354 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-11T08:23:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:13 smithi139 ceph-mon[32601]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:13 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:13 smithi139 ceph-mon[32601]: from='osd.5 v2:172.21.15.139:6804/2438260503' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:23:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:13 smithi139 ceph-mon[32601]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:23:14.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[28998]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:14.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[28998]: from='osd.5 v2:172.21.15.139:6804/2438260503' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[28998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[39404]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[39404]: from='osd.5 v2:172.21.15.139:6804/2438260503' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:23:14.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:13 smithi110 ceph-mon[39404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-11T08:23:14.519 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:14.953 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:14 smithi139 ceph-mon[32601]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:23:14.953 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:14 smithi139 ceph-mon[32601]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T08:23:14.953 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:14 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:14.953 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:14 smithi139 ceph-mon[32601]: from='osd.5 v2:172.21.15.139:6804/2438260503' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:14.954 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:14 smithi139 ceph-mon[32601]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:14.954 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:23:14 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:23:14.722+0000 7f9b08f91640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T08:23:14.954 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:23:14 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:23:14.727+0000 7f9b03dc8640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:23:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[28998]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:23:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[28998]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T08:23:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[28998]: from='osd.5 v2:172.21.15.139:6804/2438260503' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:15.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[28998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:15.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[39404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-11T08:23:15.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[39404]: osdmap e35: 6 total, 5 up, 6 in 2024-09-11T08:23:15.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:15.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[39404]: from='osd.5 v2:172.21.15.139:6804/2438260503' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:15.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:14 smithi110 ceph-mon[39404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:16.027 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:15 smithi139 ceph-mon[32601]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:16.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:15 smithi139 ceph-mon[32601]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:23:16.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:15 smithi139 ceph-mon[32601]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T08:23:16.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:16.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:16.028 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:15 smithi139 ceph-mon[32601]: osd.5 v2:172.21.15.139:6804/2438260503 boot 2024-09-11T08:23:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[28998]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[28998]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:23:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[28998]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T08:23:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[28998]: osd.5 v2:172.21.15.139:6804/2438260503 boot 2024-09-11T08:23:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[39404]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-11T08:23:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[39404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:23:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[39404]: osdmap e36: 6 total, 5 up, 6 in 2024-09-11T08:23:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:15 smithi110 ceph-mon[39404]: osd.5 v2:172.21.15.139:6804/2438260503 boot 2024-09-11T08:23:16.986 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:16 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:23:16.986 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:16 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:23:16.986 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:16 smithi139 ceph-mon[32601]: osdmap e37: 6 total, 6 up, 6 in 2024-09-11T08:23:16.986 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:16 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:16.987 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:16 smithi139 ceph-mon[32601]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T08:23:17.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:23:17.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:23:17.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[28998]: osdmap e37: 6 total, 6 up, 6 in 2024-09-11T08:23:17.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:17.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[28998]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T08:23:17.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:23:17.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:23:17.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[39404]: osdmap e37: 6 total, 6 up, 6 in 2024-09-11T08:23:17.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:23:17.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:16 smithi110 ceph-mon[39404]: osdmap e38: 6 total, 6 up, 6 in 2024-09-11T08:23:17.357 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-11T08:23:17.448 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-11T08:23:17.462 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-09-11T08:23:17.463 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-09-11T08:23:17.463 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124624 s, 841 MB/s 2024-09-11T08:23:17.463 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi139 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi139 to 8544M 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:18.055 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:18.056 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.056 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:18.056 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:18.056 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:18.056 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T08:23:18.056 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:18 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.078 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2024-09-11T08:23:18.317 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:18.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:18.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi139 2024-09-11T08:23:18.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi139 to 8544M 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T08:23:18.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi139 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi139 to 8544M 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:18.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:18.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:18.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:18.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:18.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: osdmap e39: 6 total, 6 up, 6 in 2024-09-11T08:23:18.406 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:18 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:20.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:20 smithi139 ceph-mon[32601]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:20.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:20 smithi110 ceph-mon[28998]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:20.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:20 smithi110 ceph-mon[39404]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:21.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:21 smithi139 ceph-mon[32601]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:23:21.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:21 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:23:21.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:21 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:23:21.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:21 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:21.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[28998]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:23:21.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:23:21.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:23:21.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:21.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[39404]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:23:21.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:23:21.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:23:21.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:21 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:22.137 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:22 smithi139 ceph-mon[32601]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:22.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:22 smithi110 ceph-mon[28998]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:22.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:22 smithi110 ceph-mon[39404]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:23.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:23 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/1161474439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]: dispatch 2024-09-11T08:23:23.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:23 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]: dispatch 2024-09-11T08:23:23.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:23 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]': finished 2024-09-11T08:23:23.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:23 smithi139 ceph-mon[32601]: osdmap e40: 7 total, 6 up, 7 in 2024-09-11T08:23:23.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:23 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:23.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:23 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/4158503658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/1161474439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]: dispatch 2024-09-11T08:23:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]: dispatch 2024-09-11T08:23:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]': finished 2024-09-11T08:23:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[28998]: osdmap e40: 7 total, 6 up, 7 in 2024-09-11T08:23:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:23.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/4158503658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/1161474439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]: dispatch 2024-09-11T08:23:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]: dispatch 2024-09-11T08:23:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4135e80-e46d-4edd-9066-d6f98d83936c"}]': finished 2024-09-11T08:23:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[39404]: osdmap e40: 7 total, 6 up, 7 in 2024-09-11T08:23:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:23 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/4158503658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:24.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:24 smithi139 ceph-mon[32601]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:24.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:24 smithi110 ceph-mon[28998]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:24.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:24 smithi110 ceph-mon[39404]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:26.315 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:26 smithi139 ceph-mon[32601]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:26.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:26 smithi110 ceph-mon[39404]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:26.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:26 smithi110 ceph-mon[28998]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:28.316 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:28 smithi139 ceph-mon[32601]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:28.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:28 smithi110 ceph-mon[28998]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:28.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:28 smithi110 ceph-mon[39404]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:29.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:29 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-11T08:23:29.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:29 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:29.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:29 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-11T08:23:29.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:29 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:29.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:29 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-11T08:23:29.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:29 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:30.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:30 smithi139 ceph-mon[32601]: Deploying daemon osd.6 on smithi139 2024-09-11T08:23:30.288 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:30 smithi139 ceph-mon[32601]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:30.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:30 smithi110 ceph-mon[28998]: Deploying daemon osd.6 on smithi139 2024-09-11T08:23:30.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:30 smithi110 ceph-mon[28998]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:30.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:30 smithi110 ceph-mon[39404]: Deploying daemon osd.6 on smithi139 2024-09-11T08:23:30.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:30 smithi110 ceph-mon[39404]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:32.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:32 smithi139 ceph-mon[32601]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:32.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:32 smithi110 ceph-mon[28998]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:32.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:32 smithi110 ceph-mon[39404]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:34.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:34 smithi139 ceph-mon[32601]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:34.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:34 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:34.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:34 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:34.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:34 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:34.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[39404]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:34.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:34.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:34.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:34.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[28998]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:34.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:34.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:34.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:34 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.265 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:36 smithi139 ceph-mon[32601]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:36.265 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:36 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.265 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:36 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.265 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:36 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:36.265 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:36 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:36.265 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:36 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[28998]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:36.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:36.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:36.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[39404]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:36.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:36.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:36.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:36 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:36.569 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 6 on host 'smithi139' 2024-09-11T08:23:37.216 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:37 smithi139 ceph-mon[32601]: from='osd.6 v2:172.21.15.139:6808/3113765908' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:23:37.217 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:37 smithi139 ceph-mon[32601]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:23:37.217 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:37.217 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:37.217 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:37.239 DEBUG:teuthology.orchestra.run.smithi139:osd.6> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.6.service 2024-09-11T08:23:37.242 INFO:tasks.cephadm:Deploying osd.7 on smithi139 with /dev/vg_nvme/lv_1... 2024-09-11T08:23:37.242 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-11T08:23:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[28998]: from='osd.6 v2:172.21.15.139:6808/3113765908' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:23:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[28998]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:23:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:37.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[39404]: from='osd.6 v2:172.21.15.139:6808/3113765908' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:23:37.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[39404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-11T08:23:37.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:37.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:37.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:37 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:37.959 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:38.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-mon[32601]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:38.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-mon[32601]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T08:23:38.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-mon[32601]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T08:23:38.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:38.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-mon[32601]: from='osd.6 v2:172.21.15.139:6808/3113765908' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:38.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-mon[32601]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:38.319 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:23:38.063+0000 7fbdf7a61640 -1 osd.6 0 waiting for initial osdmap 2024-09-11T08:23:38.319 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:23:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:23:38.070+0000 7fbdf2097640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:23:38.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[28998]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:38.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[28998]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T08:23:38.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[28998]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[28998]: from='osd.6 v2:172.21.15.139:6808/3113765908' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[28998]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[39404]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[39404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[39404]: osdmap e41: 7 total, 6 up, 7 in 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[39404]: from='osd.6 v2:172.21.15.139:6808/3113765908' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:38.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:38 smithi110 ceph-mon[39404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:23:39.221 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:39 smithi139 ceph-mon[32601]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:23:39.221 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:39 smithi139 ceph-mon[32601]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T08:23:39.221 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:39 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:39.221 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:39 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:39.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[28998]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:23:39.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[28998]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T08:23:39.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:39.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:39.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[39404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:23:39.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[39404]: osdmap e42: 7 total, 6 up, 7 in 2024-09-11T08:23:39.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:39.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:39 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:40.206 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:40 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:23:40.206 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:40 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:23:40.206 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:40 smithi139 ceph-mon[32601]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:40.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:40 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:40.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:40 smithi139 ceph-mon[32601]: osd.6 v2:172.21.15.139:6808/3113765908 boot 2024-09-11T08:23:40.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:40 smithi139 ceph-mon[32601]: osdmap e43: 7 total, 7 up, 7 in 2024-09-11T08:23:40.207 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:40 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:40.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:23:40.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:23:40.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[28998]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[28998]: osd.6 v2:172.21.15.139:6808/3113765908 boot 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[28998]: osdmap e43: 7 total, 7 up, 7 in 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[39404]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[39404]: osd.6 v2:172.21.15.139:6808/3113765908 boot 2024-09-11T08:23:40.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[39404]: osdmap e43: 7 total, 7 up, 7 in 2024-09-11T08:23:40.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:40 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:23:40.702 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-11T08:23:40.786 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-11T08:23:40.801 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2024-09-11T08:23:40.801 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2024-09-11T08:23:40.801 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123262 s, 851 MB/s 2024-09-11T08:23:40.801 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2024-09-11T08:23:41.360 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:41 smithi139 ceph-mon[32601]: osdmap e44: 7 total, 7 up, 7 in 2024-09-11T08:23:41.384 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi139:vg_nvme/lv_1 2024-09-11T08:23:41.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:41 smithi110 ceph-mon[28998]: osdmap e44: 7 total, 7 up, 7 in 2024-09-11T08:23:41.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:41 smithi110 ceph-mon[39404]: osdmap e44: 7 total, 7 up, 7 in 2024-09-11T08:23:41.959 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:23:42.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:42.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T08:23:42.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.327 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:42.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:42.328 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:42 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T08:23:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: osdmap e45: 7 total, 7 up, 7 in 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:23:42.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:42.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:42.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:23:42.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:42 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:43.237 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:43 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi139 2024-09-11T08:23:43.237 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:43 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi139 to 5696M 2024-09-11T08:23:43.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:43 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi139 2024-09-11T08:23:43.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:43 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi139 to 5696M 2024-09-11T08:23:43.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:43 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi139 2024-09-11T08:23:43.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:43 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi139 to 5696M 2024-09-11T08:23:44.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:44 smithi139 ceph-mon[32601]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:44 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:23:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:44 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:23:44.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:44 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[28998]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:23:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:23:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[39404]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-11T08:23:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-11T08:23:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:44 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:45.319 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:45 smithi139 ceph-mon[32601]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:23:45.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:45 smithi110 ceph-mon[28998]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:23:45.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:45 smithi110 ceph-mon[39404]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:23:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[28998]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/27501723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]: dispatch 2024-09-11T08:23:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]: dispatch 2024-09-11T08:23:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[28998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]': finished 2024-09-11T08:23:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[28998]: osdmap e46: 8 total, 7 up, 8 in 2024-09-11T08:23:46.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:23:46.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[39404]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:46.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/27501723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]: dispatch 2024-09-11T08:23:46.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]: dispatch 2024-09-11T08:23:46.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[39404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]': finished 2024-09-11T08:23:46.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[39404]: osdmap e46: 8 total, 7 up, 8 in 2024-09-11T08:23:46.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:46 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:23:46.469 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:46 smithi139 ceph-mon[32601]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:46.469 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:46 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/27501723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]: dispatch 2024-09-11T08:23:46.469 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:46 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]: dispatch 2024-09-11T08:23:46.470 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:46 smithi139 ceph-mon[32601]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "304840cf-501e-49fe-a340-f732d3a97230"}]': finished 2024-09-11T08:23:46.470 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:46 smithi139 ceph-mon[32601]: osdmap e46: 8 total, 7 up, 8 in 2024-09-11T08:23:46.470 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:46 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:23:47.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:47 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/3539247441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:47.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:47 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/3539247441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:47.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:47 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/3539247441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-11T08:23:48.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:48 smithi110 ceph-mon[28998]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:48.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:48 smithi110 ceph-mon[39404]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:48.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:48 smithi139 ceph-mon[32601]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:50.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:50 smithi139 ceph-mon[32601]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:50 smithi110 ceph-mon[28998]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:50.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:50 smithi110 ceph-mon[39404]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:52.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:52 smithi139 ceph-mon[32601]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:52.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-11T08:23:52.318 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:52 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:52 smithi110 ceph-mon[28998]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-11T08:23:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:52 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:52.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:52 smithi110 ceph-mon[39404]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-11T08:23:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:52 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:23:53.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:53 smithi110 ceph-mon[39404]: Deploying daemon osd.7 on smithi139 2024-09-11T08:23:53.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:53 smithi110 ceph-mon[28998]: Deploying daemon osd.7 on smithi139 2024-09-11T08:23:53.470 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:53 smithi139 ceph-mon[32601]: Deploying daemon osd.7 on smithi139 2024-09-11T08:23:54.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:54 smithi110 ceph-mon[28998]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:54.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:54 smithi110 ceph-mon[39404]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:54.452 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:54 smithi139 ceph-mon[32601]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:56.252 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:56 smithi139 ceph-mon[32601]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:56.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:56 smithi110 ceph-mon[28998]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:56.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:56 smithi110 ceph-mon[39404]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:57.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:57.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:57.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:57 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:57.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:57.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:57.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:57 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:57.458 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:23:57.458 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:57.458 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:57 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:23:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:23:58 smithi139 ceph-mon[32601]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:58.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:23:58 smithi110 ceph-mon[28998]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:23:58.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:23:58 smithi110 ceph-mon[39404]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:00.177 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 7 on host 'smithi139' 2024-09-11T08:24:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:00.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: from='osd.7 v2:172.21.15.139:6812/2960182441' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:24:00.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[28998]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:24:00.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:00.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:00.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:00.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: from='osd.7 v2:172.21.15.139:6812/2960182441' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:24:00.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:00 smithi110 ceph-mon[39404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:24:00.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:00.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.544 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:00.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:00.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:00.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: from='osd.7 v2:172.21.15.139:6812/2960182441' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:24:00.545 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:00 smithi139 ceph-mon[32601]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-11T08:24:00.859 DEBUG:teuthology.orchestra.run.smithi139:osd.7> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.7.service 2024-09-11T08:24:00.862 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-11T08:24:00.862 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-11T08:24:01.115 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:01.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:01.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:01.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: osdmap e47: 8 total, 7 up, 8 in 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: from='osd.7 v2:172.21.15.139:6812/2960182441' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[28998]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: osdmap e47: 8 total, 7 up, 8 in 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:01.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: from='osd.7 v2:172.21.15.139:6812/2960182441' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:24:01.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:01 smithi110 ceph-mon[39404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:24:01.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:01.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:01.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:01.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-11T08:24:01.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: osdmap e47: 8 total, 7 up, 8 in 2024-09-11T08:24:01.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:01.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: from='osd.7 v2:172.21.15.139:6812/2960182441' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:24:01.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-mon[32601]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2024-09-11T08:24:02.060 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:24:01.826+0000 7f9b82981640 -1 osd.7 0 waiting for initial osdmap 2024-09-11T08:24:02.060 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:24:01 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:24:01.832+0000 7f9b7b7b4640 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:24:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:02 smithi139 ceph-mon[32601]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:02 smithi139 ceph-mon[32601]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:24:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:02 smithi139 ceph-mon[32601]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T08:24:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:02 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:02.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:02 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:02.400 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[28998]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:02.400 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[28998]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[28998]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[39404]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[39404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[39404]: osdmap e48: 8 total, 7 up, 8 in 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:02.401 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:02 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:02.812 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:24:03.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:03 smithi139 ceph-mon[32601]: purged_snaps scrub starts 2024-09-11T08:24:03.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:03 smithi139 ceph-mon[32601]: purged_snaps scrub ok 2024-09-11T08:24:03.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:03 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4003605483' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:24:03.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:03 smithi139 ceph-mon[32601]: osd.7 v2:172.21.15.139:6812/2960182441 boot 2024-09-11T08:24:03.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:03 smithi139 ceph-mon[32601]: osdmap e49: 8 total, 8 up, 8 in 2024-09-11T08:24:03.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:03 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:03.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[28998]: purged_snaps scrub starts 2024-09-11T08:24:03.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[28998]: purged_snaps scrub ok 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4003605483' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[28998]: osd.7 v2:172.21.15.139:6812/2960182441 boot 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[28998]: osdmap e49: 8 total, 8 up, 8 in 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[39404]: purged_snaps scrub starts 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[39404]: purged_snaps scrub ok 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4003605483' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[39404]: osd.7 v2:172.21.15.139:6812/2960182441 boot 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[39404]: osdmap e49: 8 total, 8 up, 8 in 2024-09-11T08:24:03.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:03 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:03.464 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":48,"num_osds":8,"num_up_osds":7,"osd_up_since":1726043019,"num_in_osds":8,"osd_in_since":1726043025,"num_remapped_pgs":0} 2024-09-11T08:24:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:24:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[28998]: osdmap e50: 8 total, 8 up, 8 in 2024-09-11T08:24:04.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:04 smithi110 ceph-mon[39404]: osdmap e50: 8 total, 8 up, 8 in 2024-09-11T08:24:04.465 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:04.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:04.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:04.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:04 smithi139 ceph-mon[32601]: osdmap e50: 8 total, 8 up, 8 in 2024-09-11T08:24:04.712 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:05.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:05 smithi110 ceph-mon[28998]: Detected new or changed devices on smithi139 2024-09-11T08:24:05.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:05 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi139 to 4272M 2024-09-11T08:24:05.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:05 smithi110 ceph-mon[28998]: osdmap e51: 8 total, 8 up, 8 in 2024-09-11T08:24:05.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:05 smithi110 ceph-mon[39404]: Detected new or changed devices on smithi139 2024-09-11T08:24:05.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:05 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi139 to 4272M 2024-09-11T08:24:05.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:05 smithi110 ceph-mon[39404]: osdmap e51: 8 total, 8 up, 8 in 2024-09-11T08:24:05.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:05 smithi139 ceph-mon[32601]: Detected new or changed devices on smithi139 2024-09-11T08:24:05.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:05 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi139 to 4272M 2024-09-11T08:24:05.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:05 smithi139 ceph-mon[32601]: osdmap e51: 8 total, 8 up, 8 in 2024-09-11T08:24:06.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:06 smithi110 ceph-mon[39404]: pgmap v161: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:06.156 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:06 smithi110 ceph-mon[28998]: pgmap v161: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:06.555 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:24:06.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:06 smithi139 ceph-mon[32601]: pgmap v161: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:07.215 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":51,"num_osds":8,"num_up_osds":8,"osd_up_since":1726043042,"num_in_osds":8,"osd_in_since":1726043025,"num_remapped_pgs":0} 2024-09-11T08:24:07.215 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:24:07.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:07 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1489429752' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:24:07.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:07 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1489429752' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:24:07.457 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:07.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:07 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1489429752' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-11T08:24:08.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:08 smithi110 ceph-mon[28998]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:08.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:08 smithi110 ceph-mon[39404]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:08.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:08 smithi139 ceph-mon[32601]: pgmap v163: 1 pgs: 1 peering; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:09.057 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:24:09.057 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":51,"fsid":"2850e098-7016-11ef-bcea-c7b262605968","created":"2024-09-11T08:19:11.379010+0000","modified":"2024-09-11T08:24:04.825318+0000","last_up_change":"2024-09-11T08:24:02.822303+0000","last_in_change":"2024-09-11T08:23:45.961039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:22:08.796328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"76330d63-fa1a-4768-9454-1d76c578f4cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1174545642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1174545642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1174545642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":1174545642}]},"public_addr":"172.21.15.110:6801/1174545642","cluster_addr":"172.21.15.110:6802/1174545642","heartbeat_back_addr":"172.21.15.110:6804/1174545642","heartbeat_front_addr":"172.21.15.110:6803/1174545642","state":["exists","up"]},{"osd":1,"uuid":"31c2d1f0-0aa3-415c-8fc9-8e64ae860768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":1566634575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1566634575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":1566634575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":1566634575}]},"public_addr":"172.21.15.110:6805/1566634575","cluster_addr":"172.21.15.110:6806/1566634575","heartbeat_back_addr":"172.21.15.110:6808/1566634575","heartbeat_front_addr":"172.21.15.110:6807/1566634575","state":["exists","up"]},{"osd":2,"uuid":"48cdaba0-6b9f-4716-9e16-dd708fe2cc74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":2039773886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2039773886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2039773886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":2039773886}]},"public_addr":"172.21.15.110:6809/2039773886","cluster_addr":"172.21.15.110:6810/2039773886","heartbeat_back_addr":"172.21.15.110:6812/2039773886","heartbeat_front_addr":"172.21.15.110:6811/2039773886","state":["exists","up"]},{"osd":3,"uuid":"e6b86009-9a77-4998-913a-63e9c9ff1cf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":3504281989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3504281989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3504281989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":3504281989}]},"public_addr":"172.21.15.110:6813/3504281989","cluster_addr":"172.21.15.110:6814/3504281989","heartbeat_back_addr":"172.21.15.110:6816/3504281989","heartbeat_front_addr":"172.21.15.110:6815/3504281989","state":["exists","up"]},{"osd":4,"uuid":"a513f216-1a9e-4fd2-9584-f28380c9e427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2716089259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6801","nonce":2716089259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6803","nonce":2716089259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2716089259}]},"public_addr":"172.21.15.139:6800/2716089259","cluster_addr":"172.21.15.139:6801/2716089259","heartbeat_back_addr":"172.21.15.139:6803/2716089259","heartbeat_front_addr":"172.21.15.139:6802/2716089259","state":["exists","up"]},{"osd":5,"uuid":"543c8080-fb5e-4ebe-aea6-8932d7e49b96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2438260503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6805","nonce":2438260503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6807","nonce":2438260503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2438260503}]},"public_addr":"172.21.15.139:6804/2438260503","cluster_addr":"172.21.15.139:6805/2438260503","heartbeat_back_addr":"172.21.15.139:6807/2438260503","heartbeat_front_addr":"172.21.15.139:6806/2438260503","state":["exists","up"]},{"osd":6,"uuid":"a4135e80-e46d-4edd-9066-d6f98d83936c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3113765908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":3113765908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6811","nonce":3113765908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3113765908}]},"public_addr":"172.21.15.139:6808/3113765908","cluster_addr":"172.21.15.139:6809/3113765908","heartbeat_back_addr":"172.21.15.139:6811/3113765908","heartbeat_front_addr":"172.21.15.139:6810/3113765908","state":["exists","up"]},{"osd":7,"uuid":"304840cf-501e-49fe-a340-f732d3a97230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2960182441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":2960182441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6815","nonce":2960182441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2960182441}]},"public_addr":"172.21.15.139:6812/2960182441","cluster_addr":"172.21.15.139:6813/2960182441","heartbeat_back_addr":"172.21.15.139:6815/2960182441","heartbeat_front_addr":"172.21.15.139:6814/2960182441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:21:20.398495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:21:42.979949+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:05.353130+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:28.606439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:49.789777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:23:13.807176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:23:37.363108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:24:01.069396+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:6800/3278423997":"2024-09-12T08:20:02.495064+0000","172.21.15.110:0/87316770":"2024-09-12T08:20:02.495064+0000","172.21.15.110:0/4137633303":"2024-09-12T08:20:02.495064+0000","172.21.15.110:0/3827349883":"2024-09-12T08:20:02.495064+0000","172.21.15.110:6800/227906020":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/3019575362":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/4104523767":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/1563026240":"2024-09-12T08:19:40.505888+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:24:09.347 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:09 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3798916434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:09.347 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:09 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3798916434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:09.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:09 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3798916434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:09.771 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T08:22:08.796328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-11T08:24:09.771 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-11T08:24:10.010 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:10.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:10 smithi110 ceph-mon[39404]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-11T08:24:10.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:10 smithi110 ceph-mon[39404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:24:10.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:10 smithi110 ceph-mon[39404]: Cluster is now healthy 2024-09-11T08:24:10.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:10 smithi110 ceph-mon[28998]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-11T08:24:10.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:10 smithi110 ceph-mon[28998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:24:10.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:10 smithi110 ceph-mon[28998]: Cluster is now healthy 2024-09-11T08:24:10.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:10 smithi139 ceph-mon[32601]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-11T08:24:10.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:10 smithi139 ceph-mon[32601]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-11T08:24:10.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:10 smithi139 ceph-mon[32601]: Cluster is now healthy 2024-09-11T08:24:11.718 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2024-09-11T08:24:12.372 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:12 smithi110 ceph-mon[28998]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-11T08:24:12.372 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:12 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/317581729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-11T08:24:12.372 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:12 smithi110 ceph-mon[39404]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-11T08:24:12.372 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:12 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/317581729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-11T08:24:12.397 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi110 2024-09-11T08:24:12.397 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi110=foo.a' 2024-09-11T08:24:12.425 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:12 smithi139 ceph-mon[32601]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-11T08:24:12.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:12 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/317581729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-11T08:24:12.650 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:14.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:14 smithi139 ceph-mon[32601]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T08:24:14.395 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled rgw.foo.a update... 2024-09-11T08:24:14.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:14 smithi110 ceph-mon[28998]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T08:24:14.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:14 smithi110 ceph-mon[39404]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-11T08:24:14.930 DEBUG:teuthology.orchestra.run.smithi110:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@rgw.foo.a.service 2024-09-11T08:24:14.932 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi139 2024-09-11T08:24:14.933 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-11T08:24:15.219 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:15.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi110=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:15.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: Saving service rgw.foo.a spec with placement smithi110=foo.a;count:1 2024-09-11T08:24:15.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: Deploying daemon rgw.foo.a on smithi110 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[28998]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi110=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:15.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: Saving service rgw.foo.a spec with placement smithi110=foo.a;count:1 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: Deploying daemon rgw.foo.a on smithi110 2024-09-11T08:24:15.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:15 smithi110 ceph-mon[39404]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:24:15.675 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi110=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:15.675 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: Saving service rgw.foo.a spec with placement smithi110=foo.a;count:1 2024-09-11T08:24:15.675 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: Deploying daemon rgw.foo.a on smithi110 2024-09-11T08:24:15.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:15 smithi139 ceph-mon[32601]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-11T08:24:16.903 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:24:16 smithi110 systemd[1]: Started Ceph rgw.foo.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:24:17.602 INFO:teuthology.orchestra.run.smithi139.stderr:pool 'datapool' created 2024-09-11T08:24:18.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-11T08:24:18.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: Saving service rgw.foo.a spec with placement smithi110=foo.a;count:1 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/3472533805' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: osdmap e52: 8 total, 8 up, 8 in 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:24:18.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: Saving service rgw.foo.a spec with placement smithi110=foo.a;count:1 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/3472533805' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: osdmap e52: 8 total, 8 up, 8 in 2024-09-11T08:24:18.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:17 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: Saving service rgw.foo.a spec with placement smithi110=foo.a;count:1 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/3472533805' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:24:18.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-11T08:24:18.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:18.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:18.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-11T08:24:18.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: osdmap e52: 8 total, 8 up, 8 in 2024-09-11T08:24:18.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:17 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-11T08:24:18.356 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- rbd pool init datapool 2024-09-11T08:24:18.616 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:19.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:19.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:19.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: Checking dashboard <-> RGW credentials 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: pgmap v170: 36 pgs: 10 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[28998]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: Checking dashboard <-> RGW credentials 2024-09-11T08:24:19.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: pgmap v170: 36 pgs: 10 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:19.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-11T08:24:19.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:19 smithi110 ceph-mon[39404]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T08:24:19.734 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:19.734 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:19.734 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:19.734 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:19.734 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: Checking dashboard <-> RGW credentials 2024-09-11T08:24:19.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: pgmap v170: 36 pgs: 10 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:19.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-11T08:24:19.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:19 smithi139 ceph-mon[32601]: osdmap e53: 8 total, 8 up, 8 in 2024-09-11T08:24:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[39404]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:24:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[39404]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T08:24:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4136209793' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[39404]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/3664999557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:24:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:24:20.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[28998]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:24:20.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[28998]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T08:24:20.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4136209793' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[28998]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/3664999557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:24:20.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:20 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:24:20.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:20 smithi139 ceph-mon[32601]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:24:20.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:20 smithi139 ceph-mon[32601]: osdmap e54: 8 total, 8 up, 8 in 2024-09-11T08:24:20.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:20 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:20 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4136209793' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:20 smithi139 ceph-mon[32601]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-11T08:24:20.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:20 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/3664999557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:24:20.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:20 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-11T08:24:21.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[39404]: pgmap v173: 68 pgs: 13 creating+peering, 6 active+clean, 49 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:21.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:24:21.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[39404]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:24:21.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-11T08:24:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[39404]: osdmap e55: 8 total, 8 up, 8 in 2024-09-11T08:24:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[28998]: pgmap v173: 68 pgs: 13 creating+peering, 6 active+clean, 49 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:24:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[28998]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:24:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-11T08:24:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:21 smithi110 ceph-mon[28998]: osdmap e55: 8 total, 8 up, 8 in 2024-09-11T08:24:22.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:21 smithi139 ceph-mon[32601]: pgmap v173: 68 pgs: 13 creating+peering, 6 active+clean, 49 unknown; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:22.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:21 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:24:22.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:21 smithi139 ceph-mon[32601]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-11T08:24:22.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:21 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-11T08:24:22.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:21 smithi139 ceph-mon[32601]: osdmap e55: 8 total, 8 up, 8 in 2024-09-11T08:24:22.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[39404]: osdmap e56: 8 total, 8 up, 8 in 2024-09-11T08:24:22.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4136209793' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[39404]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[28998]: osdmap e56: 8 total, 8 up, 8 in 2024-09-11T08:24:22.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4136209793' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:22 smithi110 ceph-mon[28998]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.949 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:22 smithi139 ceph-mon[32601]: osdmap e56: 8 total, 8 up, 8 in 2024-09-11T08:24:22.949 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:22 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.949 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:22 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4136209793' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:22.949 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:22 smithi139 ceph-mon[32601]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-11T08:24:23.421 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.139 --placement '1;smithi139=iscsi.a' 2024-09-11T08:24:23.675 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:23.784 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:23 smithi139 ceph-mon[32601]: pgmap v176: 100 pgs: 2 creating+activating, 20 creating+peering, 52 active+clean, 26 unknown; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-11T08:24:23.784 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:23 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:24:23.784 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:23 smithi139 ceph-mon[32601]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:24:23.784 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:23 smithi139 ceph-mon[32601]: osdmap e57: 8 total, 8 up, 8 in 2024-09-11T08:24:23.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[28998]: pgmap v176: 100 pgs: 2 creating+activating, 20 creating+peering, 52 active+clean, 26 unknown; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-11T08:24:23.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:24:23.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[28998]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:24:23.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[28998]: osdmap e57: 8 total, 8 up, 8 in 2024-09-11T08:24:23.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[39404]: pgmap v176: 100 pgs: 2 creating+activating, 20 creating+peering, 52 active+clean, 26 unknown; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-11T08:24:23.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2095296519' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:24:23.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[39404]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-11T08:24:23.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:23 smithi110 ceph-mon[39404]: osdmap e57: 8 total, 8 up, 8 in 2024-09-11T08:24:24.731 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:24 smithi139 ceph-mon[32601]: osdmap e58: 8 total, 8 up, 8 in 2024-09-11T08:24:24.731 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:24 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:24:24.731 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:24 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:24:24.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:24 smithi110 ceph-mon[28998]: osdmap e58: 8 total, 8 up, 8 in 2024-09-11T08:24:24.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:24 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:24:24.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:24 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:24:24.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:24 smithi110 ceph-mon[39404]: osdmap e58: 8 total, 8 up, 8 in 2024-09-11T08:24:24.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:24 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:24:24.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:24 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-11T08:24:25.389 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled iscsi.datapool update... 2024-09-11T08:24:25.756 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:25 smithi139 ceph-mon[32601]: pgmap v179: 132 pgs: 2 creating+activating, 10 creating+peering, 72 active+clean, 48 unknown; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-11T08:24:25.756 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:25 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:24:25.756 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:25 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:24:25.756 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:25 smithi139 ceph-mon[32601]: osdmap e59: 8 total, 8 up, 8 in 2024-09-11T08:24:25.756 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:25 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:24:25.756 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:25 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:24:25.756 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:25 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:25.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[39404]: pgmap v179: 132 pgs: 2 creating+activating, 10 creating+peering, 72 active+clean, 48 unknown; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-11T08:24:25.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:24:25.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:24:25.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[39404]: osdmap e59: 8 total, 8 up, 8 in 2024-09-11T08:24:25.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:24:25.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:24:25.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:25.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[28998]: pgmap v179: 132 pgs: 2 creating+activating, 10 creating+peering, 72 active+clean, 48 unknown; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-11T08:24:25.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:24:25.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-11T08:24:25.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[28998]: osdmap e59: 8 total, 8 up, 8 in 2024-09-11T08:24:25.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:24:25.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-11T08:24:25.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:25 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.073 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-11T08:24:26.073 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:24:26.073 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-11T08:24:26.102 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:24:26.102 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-11T08:24:26.130 DEBUG:teuthology.orchestra.run.smithi139:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@iscsi.iscsi.a.service 2024-09-11T08:24:26.173 INFO:tasks.cephadm:Adding prometheus.a on smithi139 2024-09-11T08:24:26.173 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply prometheus '1;smithi139=a' 2024-09-11T08:24:26.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.139", "placement": "1;smithi139=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: Saving service iscsi.datapool spec with placement smithi139=iscsi.a;count:1 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: osdmap e60: 8 total, 8 up, 8 in 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-11T08:24:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:26.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: Deploying daemon iscsi.iscsi.a on smithi139 2024-09-11T08:24:26.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:26 smithi139 ceph-mon[32601]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:24:26.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.139", "placement": "1;smithi139=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: Saving service iscsi.datapool spec with placement smithi139=iscsi.a;count:1 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: osdmap e60: 8 total, 8 up, 8 in 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: Deploying daemon iscsi.iscsi.a on smithi139 2024-09-11T08:24:26.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[39404]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.139", "placement": "1;smithi139=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: Saving service iscsi.datapool spec with placement smithi139=iscsi.a;count:1 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2896972542' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1827275572' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: osdmap e60: 8 total, 8 up, 8 in 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-11T08:24:26.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-11T08:24:26.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:26.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: Deploying daemon iscsi.iscsi.a on smithi139 2024-09-11T08:24:26.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:26 smithi110 ceph-mon[28998]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T08:24:27.385 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:27.806 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:27 smithi139 ceph-mon[32601]: pgmap v182: 132 pgs: 7 creating+peering, 96 active+clean, 29 unknown; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-09-11T08:24:27.806 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:27 smithi139 ceph-mon[32601]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:24:27.806 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:27 smithi139 ceph-mon[32601]: Cluster is now healthy 2024-09-11T08:24:27.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:27 smithi110 ceph-mon[28998]: pgmap v182: 132 pgs: 7 creating+peering, 96 active+clean, 29 unknown; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-09-11T08:24:27.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:27 smithi110 ceph-mon[28998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:24:27.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:27 smithi110 ceph-mon[28998]: Cluster is now healthy 2024-09-11T08:24:27.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:27 smithi110 ceph-mon[39404]: pgmap v182: 132 pgs: 7 creating+peering, 96 active+clean, 29 unknown; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 KiB/s wr, 62 op/s 2024-09-11T08:24:27.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:27 smithi110 ceph-mon[39404]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-11T08:24:27.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:27 smithi110 ceph-mon[39404]: Cluster is now healthy 2024-09-11T08:24:29.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:28 smithi139 podman[59404]: 2024-09-11 08:24:28.990315448 +0000 UTC m=+0.325865139 container create 6d05d21df435000f4b718a1b8bfccd8c013de6e2c34ebb49d114f40eefc16b2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-11T08:24:29.474 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled prometheus update... 2024-09-11T08:24:29.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:29 smithi139 podman[59404]: 2024-09-11 08:24:29.41050262 +0000 UTC m=+0.746052271 container init 6d05d21df435000f4b718a1b8bfccd8c013de6e2c34ebb49d114f40eefc16b2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-11T08:24:29.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:29 smithi139 podman[59404]: 2024-09-11 08:24:29.414078483 +0000 UTC m=+0.749628135 container start 6d05d21df435000f4b718a1b8bfccd8c013de6e2c34ebb49d114f40eefc16b2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-11T08:24:29.474 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:29 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu[59442]: log file path now is '/var/log/tcmu-runner.log' 2024-09-11T08:24:29.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:29 smithi139 bash[59404]: 6d05d21df435000f4b718a1b8bfccd8c013de6e2c34ebb49d114f40eefc16b2e 2024-09-11T08:24:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:29 smithi139 ceph-mon[32601]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 6.1 KiB/s wr, 163 op/s 2024-09-11T08:24:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:29 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:29.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:29 smithi110 ceph-mon[28998]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 6.1 KiB/s wr, 163 op/s 2024-09-11T08:24:29.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:29 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:29 smithi110 ceph-mon[39404]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 6.1 KiB/s wr, 163 op/s 2024-09-11T08:24:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:29 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:30.191 DEBUG:teuthology.orchestra.run.smithi139:prometheus.a> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@prometheus.a.service 2024-09-11T08:24:30.194 INFO:tasks.cephadm:Adding node-exporter.a on smithi110 2024-09-11T08:24:30.194 INFO:tasks.cephadm:Adding node-exporter.b on smithi139 2024-09-11T08:24:30.194 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply node-exporter '2;smithi110=a;smithi139=b' 2024-09-11T08:24:30.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 podman[59527]: 2024-09-11 08:24:30.223038684 +0000 UTC m=+0.019340739 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:24:30.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 podman[59527]: 2024-09-11 08:24:30.349498811 +0000 UTC m=+0.145800859 container create 172b96c6173ad30847171680d413d3c4bfd763c838e11a34813cba7dcf4d08f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-11T08:24:30.695 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:30.902 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:30 smithi139 ceph-mon[32601]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:30.902 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:30 smithi139 ceph-mon[32601]: Saving service prometheus spec with placement smithi139=a;count:1 2024-09-11T08:24:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:30 smithi110 ceph-mon[28998]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:30 smithi110 ceph-mon[28998]: Saving service prometheus spec with placement smithi139=a;count:1 2024-09-11T08:24:30.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 podman[59527]: 2024-09-11 08:24:30.595926305 +0000 UTC m=+0.392228348 container init 172b96c6173ad30847171680d413d3c4bfd763c838e11a34813cba7dcf4d08f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-11T08:24:30.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 podman[59527]: 2024-09-11 08:24:30.599276729 +0000 UTC m=+0.395578772 container start 172b96c6173ad30847171680d413d3c4bfd763c838e11a34813cba7dcf4d08f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-11T08:24:30.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 bash[59527]: 172b96c6173ad30847171680d413d3c4bfd763c838e11a34813cba7dcf4d08f9 2024-09-11T08:24:30.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 systemd[1]: Started Ceph iscsi.iscsi.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:24:30.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug Started the configuration object watcher 2024-09-11T08:24:30.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug Checking for config object changes every 1s 2024-09-11T08:24:30.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:30 smithi110 ceph-mon[39404]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:30.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:30 smithi110 ceph-mon[39404]: Saving service prometheus spec with placement smithi139=a;count:1 2024-09-11T08:24:31.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug Processing osd blocklist entries for this node 2024-09-11T08:24:31.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug Reading the configuration object to update local LIO configuration 2024-09-11T08:24:31.187 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug Configuration does not have an entry for this host(smithi139.front.sepia.ceph.com) - nothing to define to LIO 2024-09-11T08:24:31.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-11T08:24:31.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: * Environment: production 2024-09-11T08:24:31.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-11T08:24:31.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: Use a production WSGI server instead. 2024-09-11T08:24:31.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: * Debug mode: off 2024-09-11T08:24:31.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug * Running on all addresses. 2024-09-11T08:24:31.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-11T08:24:31.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: * Running on all addresses. 2024-09-11T08:24:31.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-11T08:24:31.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug * Running on http://[fe80::ec4:7aff:fe88:7f59]:5000/ (Press CTRL+C to quit) 2024-09-11T08:24:31.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: * Running on http://[fe80::ec4:7aff:fe88:7f59]:5000/ (Press CTRL+C to quit) 2024-09-11T08:24:32.009 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 5.5 KiB/s wr, 156 op/s 2024-09-11T08:24:32.009 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.009 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.009 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.009 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-11T08:24:32.010 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.010 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:32.010 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: osdmap e61: 8 total, 8 up, 8 in 2024-09-11T08:24:32.010 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:31 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/4211795074' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-11T08:24:32.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 5.5 KiB/s wr, 156 op/s 2024-09-11T08:24:32.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: osdmap e61: 8 total, 8 up, 8 in 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/4211795074' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 5.5 KiB/s wr, 156 op/s 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-11T08:24:32.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:32.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:32.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: osdmap e61: 8 total, 8 up, 8 in 2024-09-11T08:24:32.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:31 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/4211795074' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-11T08:24:32.553 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled node-exporter update... 2024-09-11T08:24:33.687 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:33 smithi139 ceph-mon[32601]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.9 KiB/s wr, 143 op/s 2024-09-11T08:24:33.687 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:33 smithi139 ceph-mon[32601]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi110=a;smithi139=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:33.687 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:33 smithi139 ceph-mon[32601]: Saving service node-exporter spec with placement smithi110=a;smithi139=b;count:2 2024-09-11T08:24:33.687 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:33 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:33.687 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:33 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:33.710 DEBUG:teuthology.orchestra.run.smithi110:node-exporter.a> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.a.service 2024-09-11T08:24:33.713 DEBUG:teuthology.orchestra.run.smithi139:node-exporter.b> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.b.service 2024-09-11T08:24:33.715 INFO:tasks.cephadm:Adding alertmanager.a on smithi110 2024-09-11T08:24:33.715 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply alertmanager '1;smithi110=a' 2024-09-11T08:24:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[28998]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.9 KiB/s wr, 143 op/s 2024-09-11T08:24:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[28998]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi110=a;smithi139=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[28998]: Saving service node-exporter spec with placement smithi110=a;smithi139=b;count:2 2024-09-11T08:24:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[39404]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.9 KiB/s wr, 143 op/s 2024-09-11T08:24:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[39404]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi110=a;smithi139=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[39404]: Saving service node-exporter spec with placement smithi110=a;smithi139=b;count:2 2024-09-11T08:24:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:33.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:33 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:34.018 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:34.820 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:34 smithi110 ceph-mon[28998]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-11T08:24:34.820 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:34 smithi110 ceph-mon[39404]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-11T08:24:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:34 smithi139 ceph-mon[32601]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-11T08:24:35.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:35 smithi139 ceph-mon[32601]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 4.3 KiB/s wr, 126 op/s 2024-09-11T08:24:35.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:35 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:35.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:35 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:35.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:35 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:35.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:35 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:35.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:35 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:35.797 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:35 smithi139 ceph-mon[32601]: Deploying daemon prometheus.a on smithi139 2024-09-11T08:24:36.025 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[28998]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 4.3 KiB/s wr, 126 op/s 2024-09-11T08:24:36.025 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:36.025 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:36.025 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[28998]: Deploying daemon prometheus.a on smithi139 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[39404]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 4.3 KiB/s wr, 126 op/s 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:24:36.026 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:36.027 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:35 smithi110 ceph-mon[39404]: Deploying daemon prometheus.a on smithi139 2024-09-11T08:24:36.468 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled alertmanager update... 2024-09-11T08:24:37.353 DEBUG:teuthology.orchestra.run.smithi110:alertmanager.a> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@alertmanager.a.service 2024-09-11T08:24:37.356 INFO:tasks.cephadm:Adding grafana.a on smithi139 2024-09-11T08:24:37.356 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph orch apply grafana '1;smithi139=a' 2024-09-11T08:24:37.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[28998]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:37.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[28998]: Saving service alertmanager spec with placement smithi110=a;count:1 2024-09-11T08:24:37.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:37.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[28998]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 KiB/s wr, 89 op/s 2024-09-11T08:24:37.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[39404]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:37.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[39404]: Saving service alertmanager spec with placement smithi110=a;count:1 2024-09-11T08:24:37.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:37.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:37 smithi110 ceph-mon[39404]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 KiB/s wr, 89 op/s 2024-09-11T08:24:37.679 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:37.807 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:37 smithi139 ceph-mon[32601]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi110=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:37.807 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:37 smithi139 ceph-mon[32601]: Saving service alertmanager spec with placement smithi110=a;count:1 2024-09-11T08:24:37.807 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:37 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:37.807 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:37 smithi139 ceph-mon[32601]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 2.7 KiB/s wr, 89 op/s 2024-09-11T08:24:38.810 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:38 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:38.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:38 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:38.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:38 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:39.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:39 smithi139 ceph-mon[32601]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 307 B/s wr, 17 op/s 2024-09-11T08:24:39.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:39 smithi110 ceph-mon[28998]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 307 B/s wr, 17 op/s 2024-09-11T08:24:39.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:39 smithi110 ceph-mon[39404]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 307 B/s wr, 17 op/s 2024-09-11T08:24:40.048 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled grafana update... 2024-09-11T08:24:41.014 DEBUG:teuthology.orchestra.run.smithi139:grafana.a> sudo journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@grafana.a.service 2024-09-11T08:24:41.017 INFO:tasks.cephadm:Setting up client nodes... 2024-09-11T08:24:41.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-11T08:24:41.268 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:41.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:24:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:41 smithi139 ceph-mon[32601]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:41 smithi139 ceph-mon[32601]: Saving service grafana spec with placement smithi139=a;count:1 2024-09-11T08:24:41.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:41 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:41.388 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:41 smithi110 ceph-mon[28998]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:41.388 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:41 smithi110 ceph-mon[28998]: Saving service grafana spec with placement smithi139=a;count:1 2024-09-11T08:24:41.388 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:41 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:41.389 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:41 smithi110 ceph-mon[39404]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi139=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:24:41.389 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:41 smithi110 ceph-mon[39404]: Saving service grafana spec with placement smithi139=a;count:1 2024-09-11T08:24:41.389 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:41 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:42.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:42 smithi139 ceph-mon[32601]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 102 B/s wr, 5 op/s 2024-09-11T08:24:42.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:42 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:42 smithi110 ceph-mon[28998]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 102 B/s wr, 5 op/s 2024-09-11T08:24:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:42 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:42.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:42 smithi110 ceph-mon[39404]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 102 B/s wr, 5 op/s 2024-09-11T08:24:42.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:42 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:43.134 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-09-11T08:24:43.134 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDLU+Fmj3jCBxAAjdxMeUQqqDhF1RN56NciSw== 2024-09-11T08:24:43.819 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-11T08:24:43.820 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-11T08:24:43.820 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T08:24:43.858 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-11T08:24:44.141 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:44 smithi139 ceph-mon[32601]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-11T08:24:44.141 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:44 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2188510786' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:44.141 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:44 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:44.141 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:44 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:24:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[28998]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-11T08:24:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2188510786' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:24:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[39404]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-11T08:24:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2188510786' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:44 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:24:44.569 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.b/config 2024-09-11T08:24:46.259 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:46 smithi139 ceph-mon[32601]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:24:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:46 smithi110 ceph-mon[28998]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:24:46.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:46 smithi110 ceph-mon[39404]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:24:48.059 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:47 smithi139 systemd[1]: Starting Ceph prometheus.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:24:48.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:48 smithi139 ceph-mon[32601]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:24:48.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:48 smithi110 ceph-mon[28998]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:24:48.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:48 smithi110 ceph-mon[39404]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:24:48.755 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:48 smithi139 podman[61113]: 2024-09-11 08:24:48.521396406 +0000 UTC m=+0.197133056 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-11T08:24:48.755 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:48 smithi139 podman[61113]: 2024-09-11 08:24:48.630090332 +0000 UTC m=+0.305826973 container create 7edc207477be36ddef2fc969977510017e8813fadfe5563cd80d32e0ea010adc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:24:49.193 INFO:teuthology.orchestra.run.smithi139.stdout:[client.1] 2024-09-11T08:24:49.194 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDRU+Fm7oMxCxAAeUGblaSm29Vj/tK+ItDMaA== 2024-09-11T08:24:49.470 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 podman[61113]: 2024-09-11 08:24:49.174979661 +0000 UTC m=+0.850716302 container init 7edc207477be36ddef2fc969977510017e8813fadfe5563cd80d32e0ea010adc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 podman[61113]: 2024-09-11 08:24:49.177864158 +0000 UTC m=+0.853600802 container start 7edc207477be36ddef2fc969977510017e8813fadfe5563cd80d32e0ea010adc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.202Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.202Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.202Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi139 (none))" 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.202Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.202Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.203Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.204Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.204Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.205Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-11T08:24:49.471 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.207Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.207Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.999µs 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.207Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.208Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.208Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.561µs wal_replay_duration=544.339µs wbl_replay_duration=130ns total_replay_duration=590.951µs 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.209Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.209Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.209Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.226Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.533571ms db_storage=870ns remote_storage=1.295µs web_handler=394ns query_engine=538ns scrape=367.955µs scrape_sd=17.594µs notify=733ns notify_sd=1.344µs rules=15.934677ms tracing=10.599µs 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.226Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:24:49.226Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 bash[61113]: 7edc207477be36ddef2fc969977510017e8813fadfe5563cd80d32e0ea010adc 2024-09-11T08:24:49.472 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:24:49 smithi139 systemd[1]: Started Ceph prometheus.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:24:50.087 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-11T08:24:50.087 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-11T08:24:50.088 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-11T08:24:50.125 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T08:24:50.125 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-11T08:24:50.126 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-11T08:24:50.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:24:50.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: from='client.? 172.21.15.139:0/4213245304' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:50.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:50.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:24:50.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-11T08:24:50.375 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: from='client.? 172.21.15.139:0/4213245304' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: from='client.? 172.21.15.139:0/4213245304' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' 2024-09-11T08:24:50.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-11T08:24:50.695 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ignoring --setuser ceph since I am not root 2024-09-11T08:24:50.695 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ignoring --setgroup ceph since I am not root 2024-09-11T08:24:50.695 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-mgr[34437]: -- 172.21.15.139:0/1937409834 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ae495ac340 con 0x55ae495b8800 2024-09-11T08:24:50.695 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:50.572+0000 7f8084ee3ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:24:50.695 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:50.628+0000 7f8084ee3ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:24:50.695 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:50.693+0000 7f8084ee3ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:24:50.716 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setuser ceph since I am not root 2024-09-11T08:24:50.716 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setgroup ceph since I am not root 2024-09-11T08:24:50.716 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-mgr[29229]: -- 172.21.15.110:0/1605355969 <== mon.0 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5609fda72340 con 0x5609fda7e800 2024-09-11T08:24:50.717 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:50.575+0000 7f36380d9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:24:50.717 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:50.636+0000 7f36380d9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:24:50.717 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:50.702+0000 7f36380d9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:24:50.954 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:50.817+0000 7f8084ee3ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:24:50.966 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:50.830+0000 7f36380d9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:24:51.207 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:50 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:50.953+0000 7f8084ee3ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:24:51.225 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:50 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:50.966+0000 7f36380d9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:24:51.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:51 smithi139 ceph-mon[32601]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-11T08:24:51.486 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:51 smithi139 ceph-mon[32601]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-11T08:24:51.486 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:51 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:51.206+0000 7f8084ee3ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:24:51.486 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:51 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:51.265+0000 7f8084ee3ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:24:51.487 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:51 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:51.424+0000 7f8084ee3ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:24:51.491 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:51 smithi110 ceph-mon[28998]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-11T08:24:51.491 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:51 smithi110 ceph-mon[28998]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-11T08:24:51.491 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:51 smithi110 ceph-mon[39404]: from='mgr.14150 172.21.15.110:0/3560243819' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-11T08:24:51.491 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:51 smithi110 ceph-mon[39404]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-11T08:24:51.491 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:51 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:51.224+0000 7f36380d9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:24:51.491 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:51 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:51.286+0000 7f36380d9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:24:51.492 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:51 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:51.448+0000 7f36380d9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:24:52.279 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:24:52.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:52 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3237671732' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-11T08:24:52.572 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:52 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:52.268+0000 7f8084ee3ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:24:52.572 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:52 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:52.333+0000 7f8084ee3ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:24:52.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:52 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3237671732' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-11T08:24:52.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:52 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3237671732' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-11T08:24:52.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:52 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:52.336+0000 7f36380d9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:24:52.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:52 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:52.402+0000 7f36380d9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:24:52.971 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":3373788203}]},"active_addr":"172.21.15.110:6800/3373788203","active_change":"2024-09-11T08:20:02.495213+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.110:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":2233151807}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1114337924}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":984481648}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":3098666162}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":2560421}]}]} 2024-09-11T08:24:52.976 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-11T08:24:52.976 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-11T08:24:52.976 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:24:53.229 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.127+0000 7f36380d9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:24:53.241 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:53.321 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.038+0000 7f8084ee3ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:24:53.322 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.150+0000 7f8084ee3ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:24:53.322 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.269+0000 7f8084ee3ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:24:53.482 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.261+0000 7f36380d9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:24:53.482 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.384+0000 7f36380d9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:24:53.482 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.444+0000 7f36380d9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:24:53.711 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.328+0000 7f8084ee3ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:24:53.712 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.451+0000 7f8084ee3ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:24:53.712 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.519+0000 7f8084ee3ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:24:53.712 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.576+0000 7f8084ee3ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:24:53.712 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.710+0000 7f8084ee3ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:24:53.838 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.573+0000 7f36380d9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:24:53.838 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.644+0000 7f36380d9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:24:53.838 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.702+0000 7f36380d9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:24:53.838 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.837+0000 7f36380d9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:24:53.977 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.773+0000 7f8084ee3ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:24:53.977 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.838+0000 7f8084ee3ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:24:53.977 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.918+0000 7f8084ee3ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:24:54.109 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.906+0000 7f36380d9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:24:54.109 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:53 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:53.971+0000 7f36380d9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:24:54.109 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:54.051+0000 7f36380d9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:24:54.109 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:54.108+0000 7f36380d9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:24:54.321 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:53.976+0000 7f8084ee3ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:24:54.322 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:54.032+0000 7f8084ee3ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:24:54.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:54.167+0000 7f36380d9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:24:55.022 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:54.950+0000 7f36380d9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[39404]: Standby manager daemon x restarted 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[39404]: Standby manager daemon x started 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[28998]: Standby manager daemon x restarted 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[28998]: Standby manager daemon x started 2024-09-11T08:24:55.023 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:24:55.024 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:24:55.024 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:24:55.024 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:54 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:24:55.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-mon[32601]: Standby manager daemon x restarted 2024-09-11T08:24:55.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-mon[32601]: Standby manager daemon x started 2024-09-11T08:24:55.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:24:55.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:24:55.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:24:55.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:24:55.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:54.777+0000 7f8084ee3ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:24:55.073 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:24:54] ENGINE Bus STARTING 2024-09-11T08:24:55.073 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: CherryPy Checker: 2024-09-11T08:24:55.073 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: The Application mounted at '' has an empty config. 2024-09-11T08:24:55.073 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:24:55.073 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:24:54] ENGINE Serving on http://:::9283 2024-09-11T08:24:55.073 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:24:54 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:24:54] ENGINE Bus STARTED 2024-09-11T08:24:55.139 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:24:55.139 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":62,"fsid":"2850e098-7016-11ef-bcea-c7b262605968","created":"2024-09-11T08:19:11.379010+0000","modified":"2024-09-11T08:24:54.952717+0000","last_up_change":"2024-09-11T08:24:02.822303+0000","last_in_change":"2024-09-11T08:23:45.961039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:22:08.796328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-11T08:24:16.917537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-11T08:24:17.174846+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-11T08:24:18.881269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-11T08:24:20.852233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-11T08:24:22.920214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"76330d63-fa1a-4768-9454-1d76c578f4cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1174545642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1174545642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1174545642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":1174545642}]},"public_addr":"172.21.15.110:6801/1174545642","cluster_addr":"172.21.15.110:6802/1174545642","heartbeat_back_addr":"172.21.15.110:6804/1174545642","heartbeat_front_addr":"172.21.15.110:6803/1174545642","state":["exists","up"]},{"osd":1,"uuid":"31c2d1f0-0aa3-415c-8fc9-8e64ae860768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":1566634575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1566634575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":1566634575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":1566634575}]},"public_addr":"172.21.15.110:6805/1566634575","cluster_addr":"172.21.15.110:6806/1566634575","heartbeat_back_addr":"172.21.15.110:6808/1566634575","heartbeat_front_addr":"172.21.15.110:6807/1566634575","state":["exists","up"]},{"osd":2,"uuid":"48cdaba0-6b9f-4716-9e16-dd708fe2cc74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":2039773886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2039773886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2039773886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":2039773886}]},"public_addr":"172.21.15.110:6809/2039773886","cluster_addr":"172.21.15.110:6810/2039773886","heartbeat_back_addr":"172.21.15.110:6812/2039773886","heartbeat_front_addr":"172.21.15.110:6811/2039773886","state":["exists","up"]},{"osd":3,"uuid":"e6b86009-9a77-4998-913a-63e9c9ff1cf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":3504281989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3504281989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3504281989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":3504281989}]},"public_addr":"172.21.15.110:6813/3504281989","cluster_addr":"172.21.15.110:6814/3504281989","heartbeat_back_addr":"172.21.15.110:6816/3504281989","heartbeat_front_addr":"172.21.15.110:6815/3504281989","state":["exists","up"]},{"osd":4,"uuid":"a513f216-1a9e-4fd2-9584-f28380c9e427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2716089259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6801","nonce":2716089259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6803","nonce":2716089259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2716089259}]},"public_addr":"172.21.15.139:6800/2716089259","cluster_addr":"172.21.15.139:6801/2716089259","heartbeat_back_addr":"172.21.15.139:6803/2716089259","heartbeat_front_addr":"172.21.15.139:6802/2716089259","state":["exists","up"]},{"osd":5,"uuid":"543c8080-fb5e-4ebe-aea6-8932d7e49b96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2438260503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6805","nonce":2438260503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6807","nonce":2438260503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2438260503}]},"public_addr":"172.21.15.139:6804/2438260503","cluster_addr":"172.21.15.139:6805/2438260503","heartbeat_back_addr":"172.21.15.139:6807/2438260503","heartbeat_front_addr":"172.21.15.139:6806/2438260503","state":["exists","up"]},{"osd":6,"uuid":"a4135e80-e46d-4edd-9066-d6f98d83936c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3113765908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":3113765908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6811","nonce":3113765908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3113765908}]},"public_addr":"172.21.15.139:6808/3113765908","cluster_addr":"172.21.15.139:6809/3113765908","heartbeat_back_addr":"172.21.15.139:6811/3113765908","heartbeat_front_addr":"172.21.15.139:6810/3113765908","state":["exists","up"]},{"osd":7,"uuid":"304840cf-501e-49fe-a340-f732d3a97230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2960182441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":2960182441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6815","nonce":2960182441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2960182441}]},"public_addr":"172.21.15.139:6812/2960182441","cluster_addr":"172.21.15.139:6813/2960182441","heartbeat_back_addr":"172.21.15.139:6815/2960182441","heartbeat_front_addr":"172.21.15.139:6814/2960182441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:21:20.398495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:21:42.979949+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:05.353130+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:28.606439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:49.789777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:23:13.807176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:23:37.363108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:24:01.069396+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:0/3098666162":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/1563026240":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/4104523767":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/3019575362":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/1114337924":"2024-09-12T08:24:54.952686+0000","172.21.15.110:6800/227906020":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/3827349883":"2024-09-12T08:20:02.495064+0000","172.21.15.110:0/4137633303":"2024-09-12T08:20:02.495064+0000","172.21.15.110:0/87316770":"2024-09-12T08:20:02.495064+0000","172.21.15.110:6800/3278423997":"2024-09-12T08:20:02.495064+0000","172.21.15.110:6800/3373788203":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/984481648":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/2233151807":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/2560421":"2024-09-12T08:24:54.952686+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:24:55.274 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:24:55] ENGINE Bus STARTING 2024-09-11T08:24:55.275 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: CherryPy Checker: 2024-09-11T08:24:55.275 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: The Application mounted at '' has an empty config. 2024-09-11T08:24:55.275 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:55.275 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:24:55] ENGINE Serving on http://:::9283 2024-09-11T08:24:55.275 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:24:55] ENGINE Bus STARTED 2024-09-11T08:24:55.805 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-11T08:24:55.806 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-11T08:24:55.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-11T08:24:55.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: Active manager daemon y restarted 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: Activating manager daemon y 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: osdmap e62: 8 total, 8 up, 8 in 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: mgrmap e19: y(active, starting, since 0.00970055s), standbys: x 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:24:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: Manager daemon y is now available 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:24:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/465678840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: Active manager daemon y restarted 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: Activating manager daemon y 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: osdmap e62: 8 total, 8 up, 8 in 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: mgrmap e19: y(active, starting, since 0.00970055s), standbys: x 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:24:55.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:24:55.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: Manager daemon y is now available 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:24:55.908 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:55 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/465678840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:56.102 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: Active manager daemon y restarted 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: Activating manager daemon y 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: osdmap e62: 8 total, 8 up, 8 in 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: mgrmap e19: y(active, starting, since 0.00970055s), standbys: x 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:24:56.103 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:24:56.104 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: Manager daemon y is now available 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:24:56.105 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:55 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/465678840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:56.732 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:57.147 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:56 smithi139 ceph-mon[32601]: mgrmap e20: y(active, since 1.01316s), standbys: x 2024-09-11T08:24:57.147 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:56 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:57.147 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:56 smithi139 ceph-mon[32601]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:57.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:56 smithi110 ceph-mon[28998]: mgrmap e20: y(active, since 1.01316s), standbys: x 2024-09-11T08:24:57.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:56 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:57.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:56 smithi110 ceph-mon[28998]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:57.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:56 smithi110 ceph-mon[39404]: mgrmap e20: y(active, since 1.01316s), standbys: x 2024-09-11T08:24:57.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:56 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:24:57.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:56 smithi110 ceph-mon[39404]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:24:57.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:57.410+0000 7f35d6e53640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:24:57] ENGINE Error in HTTPServer.serve 2024-09-11T08:24:57.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: Traceback (most recent call last): 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._connections.run(self.expiration_interval) 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._run(expiration_interval) 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: s = self.context.wrap_socket( 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: return self.sslsocket_class._create( 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:24:57.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self.do_handshake() 2024-09-11T08:24:57.655 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:24:57.655 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: self._sslobj.do_handshake() 2024-09-11T08:24:57.655 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:24:57.655 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:24:58.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[28998]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-11T08:24:58.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[28998]: [11/Sep/2024:08:24:57] ENGINE Bus STARTING 2024-09-11T08:24:58.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[28998]: [11/Sep/2024:08:24:57] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:24:58.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[28998]: [11/Sep/2024:08:24:57] ENGINE Bus STARTED 2024-09-11T08:24:58.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[28998]: [11/Sep/2024:08:24:57] ENGINE Error in HTTPServer.serve 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:24:58.154 INFO:journalctl@ceph.mon.a.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.a.smithi110.stdout: self.do_handshake() 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.a.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[39404]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[39404]: [11/Sep/2024:08:24:57] ENGINE Bus STARTING 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[39404]: [11/Sep/2024:08:24:57] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[39404]: [11/Sep/2024:08:24:57] ENGINE Bus STARTED 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[39404]: [11/Sep/2024:08:24:57] ENGINE Error in HTTPServer.serve 2024-09-11T08:24:58.155 INFO:journalctl@ceph.mon.c.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:24:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout: self.do_handshake() 2024-09-11T08:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:57 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:58.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:57 smithi139 ceph-mon[32601]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:57 smithi139 ceph-mon[32601]: [11/Sep/2024:08:24:57] ENGINE Bus STARTING 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:57 smithi139 ceph-mon[32601]: [11/Sep/2024:08:24:57] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:57 smithi139 ceph-mon[32601]: [11/Sep/2024:08:24:57] ENGINE Bus STARTED 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:57 smithi139 ceph-mon[32601]: [11/Sep/2024:08:24:57] ENGINE Error in HTTPServer.serve 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: Traceback (most recent call last): 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._run(expiration_interval) 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:24:58.322 INFO:journalctl@ceph.mon.b.smithi139.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: s = self.context.wrap_socket( 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: return self.sslsocket_class._create( 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: self.do_handshake() 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._sslobj.do_handshake() 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:24:58.323 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:57 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:58.574 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:24:58.575 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":62,"fsid":"2850e098-7016-11ef-bcea-c7b262605968","created":"2024-09-11T08:19:11.379010+0000","modified":"2024-09-11T08:24:54.952717+0000","last_up_change":"2024-09-11T08:24:02.822303+0000","last_in_change":"2024-09-11T08:23:45.961039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:22:08.796328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-11T08:24:16.917537+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-11T08:24:17.174846+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-11T08:24:18.881269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-11T08:24:20.852233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-11T08:24:22.920214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"76330d63-fa1a-4768-9454-1d76c578f4cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":1174545642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1174545642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1174545642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":1174545642}]},"public_addr":"172.21.15.110:6801/1174545642","cluster_addr":"172.21.15.110:6802/1174545642","heartbeat_back_addr":"172.21.15.110:6804/1174545642","heartbeat_front_addr":"172.21.15.110:6803/1174545642","state":["exists","up"]},{"osd":1,"uuid":"31c2d1f0-0aa3-415c-8fc9-8e64ae860768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":1566634575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1566634575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":1566634575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":1566634575}]},"public_addr":"172.21.15.110:6805/1566634575","cluster_addr":"172.21.15.110:6806/1566634575","heartbeat_back_addr":"172.21.15.110:6808/1566634575","heartbeat_front_addr":"172.21.15.110:6807/1566634575","state":["exists","up"]},{"osd":2,"uuid":"48cdaba0-6b9f-4716-9e16-dd708fe2cc74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":2039773886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":2039773886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":2039773886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":2039773886}]},"public_addr":"172.21.15.110:6809/2039773886","cluster_addr":"172.21.15.110:6810/2039773886","heartbeat_back_addr":"172.21.15.110:6812/2039773886","heartbeat_front_addr":"172.21.15.110:6811/2039773886","state":["exists","up"]},{"osd":3,"uuid":"e6b86009-9a77-4998-913a-63e9c9ff1cf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":3504281989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3504281989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3504281989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":3504281989}]},"public_addr":"172.21.15.110:6813/3504281989","cluster_addr":"172.21.15.110:6814/3504281989","heartbeat_back_addr":"172.21.15.110:6816/3504281989","heartbeat_front_addr":"172.21.15.110:6815/3504281989","state":["exists","up"]},{"osd":4,"uuid":"a513f216-1a9e-4fd2-9584-f28380c9e427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2716089259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6801","nonce":2716089259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6803","nonce":2716089259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2716089259}]},"public_addr":"172.21.15.139:6800/2716089259","cluster_addr":"172.21.15.139:6801/2716089259","heartbeat_back_addr":"172.21.15.139:6803/2716089259","heartbeat_front_addr":"172.21.15.139:6802/2716089259","state":["exists","up"]},{"osd":5,"uuid":"543c8080-fb5e-4ebe-aea6-8932d7e49b96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2438260503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6805","nonce":2438260503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6807","nonce":2438260503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2438260503}]},"public_addr":"172.21.15.139:6804/2438260503","cluster_addr":"172.21.15.139:6805/2438260503","heartbeat_back_addr":"172.21.15.139:6807/2438260503","heartbeat_front_addr":"172.21.15.139:6806/2438260503","state":["exists","up"]},{"osd":6,"uuid":"a4135e80-e46d-4edd-9066-d6f98d83936c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3113765908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6809","nonce":3113765908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6811","nonce":3113765908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3113765908}]},"public_addr":"172.21.15.139:6808/3113765908","cluster_addr":"172.21.15.139:6809/3113765908","heartbeat_back_addr":"172.21.15.139:6811/3113765908","heartbeat_front_addr":"172.21.15.139:6810/3113765908","state":["exists","up"]},{"osd":7,"uuid":"304840cf-501e-49fe-a340-f732d3a97230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":2960182441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6813","nonce":2960182441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6815","nonce":2960182441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":2960182441}]},"public_addr":"172.21.15.139:6812/2960182441","cluster_addr":"172.21.15.139:6813/2960182441","heartbeat_back_addr":"172.21.15.139:6815/2960182441","heartbeat_front_addr":"172.21.15.139:6814/2960182441","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:21:20.398495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:21:42.979949+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:05.353130+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:28.606439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:22:49.789777+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:23:13.807176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:23:37.363108+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:24:01.069396+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.110:0/3098666162":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/1563026240":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/4104523767":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/3019575362":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/1114337924":"2024-09-12T08:24:54.952686+0000","172.21.15.110:6800/227906020":"2024-09-12T08:19:40.505888+0000","172.21.15.110:0/3827349883":"2024-09-12T08:20:02.495064+0000","172.21.15.110:0/4137633303":"2024-09-12T08:20:02.495064+0000","172.21.15.110:0/87316770":"2024-09-12T08:20:02.495064+0000","172.21.15.110:6800/3278423997":"2024-09-12T08:20:02.495064+0000","172.21.15.110:6800/3373788203":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/984481648":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/2233151807":"2024-09-12T08:24:54.952686+0000","172.21.15.110:0/2560421":"2024-09-12T08:24:54.952686+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:24:59.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2912560736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2912560736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:24:59 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-11T08:24:59.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-11T08:24:59.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-11T08:24:59.336 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-11T08:24:59.336 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-11T08:24:59.337 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-11T08:24:59.337 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-11T08:24:59.338 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-11T08:24:59.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2912560736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:24:59.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:24:59 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:24:59.811 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:59.953 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:59.954 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:59.954 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:59.958 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:59.958 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:59.958 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:24:59.960 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi139 to 3760M 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi110 to 3760M 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi139 to 3760M 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.588 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi110 to 3760M 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.589 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi139 to 3760M 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi110 to 3760M 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:00.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.733 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:25:01.733 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:25:01.733 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:25:01.734 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:25:01.735 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:25:01.735 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.735 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.735 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:01 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:01 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:02.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:02 smithi110 ceph-mon[28998]: Deploying daemon alertmanager.a on smithi110 2024-09-11T08:25:02.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:02 smithi110 ceph-mon[28998]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:25:02.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:02 smithi110 ceph-mon[39404]: Deploying daemon alertmanager.a on smithi110 2024-09-11T08:25:02.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:02 smithi110 ceph-mon[39404]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:25:02.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:02 smithi139 ceph-mon[32601]: Deploying daemon alertmanager.a on smithi110 2024-09-11T08:25:02.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:02 smithi139 ceph-mon[32601]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:25:04.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:25:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:25:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:04.757 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:04 smithi110 ceph-mon[28998]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:25:04.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:04 smithi110 ceph-mon[39404]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:25:04.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:04 smithi139 ceph-mon[32601]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:25:06.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:25:05 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:25:06.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:06 smithi110 ceph-mon[28998]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:25:06.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:06 smithi110 ceph-mon[39404]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:25:06.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:06 smithi139 ceph-mon[32601]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:25:07.542 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:07 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:07.542 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:07 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:07.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:07 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:08.603 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:08 smithi110 ceph-mon[39404]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-11T08:25:08.603 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:08 smithi110 ceph-mon[28998]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-11T08:25:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:08 smithi139 ceph-mon[32601]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-11T08:25:08.928 INFO:teuthology.orchestra.run.smithi110.stdout:77309411367 2024-09-11T08:25:08.929 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-11T08:25:10.701 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:10 smithi110 ceph-mon[28998]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T08:25:10.701 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:10.701 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:10 smithi110 ceph-mon[39404]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T08:25:10.701 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:10.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:10 smithi139 ceph-mon[32601]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T08:25:10.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:11.440 INFO:teuthology.orchestra.run.smithi110.stdout:34359738414 2024-09-11T08:25:11.440 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-11T08:25:12.814 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:12 smithi110 ceph-mon[39404]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T08:25:12.815 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:12 smithi110 ceph-mon[28998]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T08:25:12.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:12 smithi139 ceph-mon[32601]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-11T08:25:13.181 INFO:teuthology.orchestra.run.smithi110.stdout:158913789975 2024-09-11T08:25:13.182 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-11T08:25:14.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:25:13 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:25:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:14.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:14 smithi139 ceph-mon[32601]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-11T08:25:14.834 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:14 smithi110 ceph-mon[28998]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-11T08:25:14.835 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:14 smithi110 ceph-mon[39404]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-11T08:25:15.112 INFO:teuthology.orchestra.run.smithi110.stdout:55834574892 2024-09-11T08:25:15.113 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-11T08:25:16.121 INFO:teuthology.orchestra.run.smithi110.stdout:184683593749 2024-09-11T08:25:16.122 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-11T08:25:16.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:25:15 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:25:16.821 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:16 smithi110 ceph-mon[39404]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:25:16.821 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:16 smithi110 ceph-mon[28998]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:25:16.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:16 smithi139 ceph-mon[32601]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:25:16.843 INFO:teuthology.orchestra.run.smithi110.stdout:111669149731 2024-09-11T08:25:16.843 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-11T08:25:17.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:17.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:18.733 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:18 smithi110 ceph-mon[28998]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:18.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:18 smithi110 ceph-mon[39404]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:18.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:18 smithi139 ceph-mon[32601]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:19.596 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:19.597 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:19.598 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:20.639 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:20 smithi110 ceph-mon[28998]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:20.640 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:20 smithi110 ceph-mon[39404]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:20.641 INFO:teuthology.orchestra.run.smithi110.stdout:133143986207 2024-09-11T08:25:20.641 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-11T08:25:20.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:20 smithi139 ceph-mon[32601]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:22.032 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:22.032 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:22.702 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:22 smithi110 ceph-mon[28998]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:22.703 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:22 smithi110 ceph-mon[39404]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:22.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:22 smithi139 ceph-mon[32601]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:23.289 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:25:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:25:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:24.546 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:24 smithi110 ceph-mon[28998]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:24.546 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:24 smithi110 ceph-mon[39404]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:24.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:24 smithi139 ceph-mon[32601]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:25 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:25.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:25 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:25.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:25 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:26.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:25:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:25:26.721 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:26 smithi110 ceph-mon[28998]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:26.721 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:26 smithi110 ceph-mon[39404]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:26.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:26 smithi139 ceph-mon[32601]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:27.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:27.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:27.779 INFO:teuthology.orchestra.run.smithi110.stdout:210453397522 2024-09-11T08:25:27.780 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-11T08:25:27.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:28.465 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:25:28 smithi110 systemd[1]: Started Ceph alertmanager.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:25:28.721 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[28998]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:25:28.721 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.721 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.721 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.721 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.721 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:25:28] ENGINE Bus STOPPING 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:25:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:25:28] ENGINE Bus STOPPED 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:25:28] ENGINE Bus STARTING 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[39404]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.722 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:28 smithi139 ceph-mon[32601]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-11T08:25:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:28 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:28 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:28 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:28 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:28.996 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:25:28] ENGINE Serving on http://:::9283 2024-09-11T08:25:28.996 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:25:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:25:28] ENGINE Bus STARTED 2024-09-11T08:25:29.463 INFO:teuthology.orchestra.run.smithi110.stdout:77309411371 2024-09-11T08:25:29.634 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[39404]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1421413787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[28998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.635 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:29 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1421413787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:25:29.682 INFO:teuthology.orchestra.run.smithi110.stdout:158913789979 2024-09-11T08:25:29.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:29 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:29 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:29 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:25:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:29 smithi139 ceph-mon[32601]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-11T08:25:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:29 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:29 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1421413787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-11T08:25:29.882 INFO:teuthology.orchestra.run.smithi110.stdout:55834574895 2024-09-11T08:25:30.503 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[28998]: Deploying daemon grafana.a on smithi139 2024-09-11T08:25:30.503 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[28998]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:30.503 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1514820975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:25:30.504 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3217232195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:25:30.504 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:30.504 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[39404]: Deploying daemon grafana.a on smithi139 2024-09-11T08:25:30.504 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[39404]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:30.504 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1514820975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:25:30.504 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3217232195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:25:30.504 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:30.505 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:25:30 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[71483]: ts=2024-09-11T08:25:30.292Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000306267s 2024-09-11T08:25:30.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789975 got 158913789979 for osd.5 2024-09-11T08:25:30.506 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:30.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:30 smithi139 ceph-mon[32601]: Deploying daemon grafana.a on smithi139 2024-09-11T08:25:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:30 smithi139 ceph-mon[32601]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:30 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1514820975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-11T08:25:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:30 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3217232195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-11T08:25:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:30 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:31.129 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411371 for osd.2 2024-09-11T08:25:31.129 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:31.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574895 for osd.1 2024-09-11T08:25:31.524 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:32.607 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:32 smithi110 ceph-mon[39404]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:32.608 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:32 smithi110 ceph-mon[28998]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:32.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:32 smithi139 ceph-mon[32601]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:34.009 INFO:teuthology.orchestra.run.smithi110.stdout:184683593754 2024-09-11T08:25:34.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:25:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:25:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:34.191 INFO:teuthology.orchestra.run.smithi110.stdout:111669149735 2024-09-11T08:25:34.709 INFO:teuthology.orchestra.run.smithi110.stdout:34359738419 2024-09-11T08:25:34.719 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:34 smithi110 ceph-mon[28998]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:34.719 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:34 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1203020538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:25:34.719 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:34 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1525036862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:25:34.720 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:34 smithi110 ceph-mon[39404]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:34.720 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:34 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1203020538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:25:34.720 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:34 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1525036862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:25:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:34 smithi139 ceph-mon[32601]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:34 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1203020538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-11T08:25:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:34 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1525036862' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-11T08:25:35.025 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:35.051 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149731 got 111669149735 for osd.3 2024-09-11T08:25:35.051 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:35 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2090188113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:25:35.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:35 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2090188113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:25:35.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:35 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2090188113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-11T08:25:36.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:25:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:25:36.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:36 smithi139 ceph-mon[32601]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:36.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:36 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:36.603 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:36 smithi110 ceph-mon[39404]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:36.603 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:36 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:36.604 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:36 smithi110 ceph-mon[28998]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:36.604 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:36 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:36.807 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738419 for osd.0 2024-09-11T08:25:36.807 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:37.272 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593749 got 184683593754 for osd.6 2024-09-11T08:25:37.272 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:37.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:37 smithi139 ceph-mon[32601]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:37.834 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:37 smithi110 ceph-mon[28998]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:37.835 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:37 smithi110 ceph-mon[39404]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:38.240 INFO:teuthology.orchestra.run.smithi110.stdout:133143986211 2024-09-11T08:25:38.403 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:25:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[71483]: ts=2024-09-11T08:25:38.294Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00205234s 2024-09-11T08:25:38.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:38 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3285668171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:25:38.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:38 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3285668171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:25:38.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:38 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3285668171' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-11T08:25:39.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986207 got 133143986211 for osd.4 2024-09-11T08:25:39.041 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:39.072 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:38 smithi139 systemd[1]: Starting Ceph grafana.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:25:39.379 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 podman[63026]: 2024-09-11 08:25:39.023273138 +0000 UTC m=+0.014842616 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-11T08:25:39.379 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 podman[63026]: 2024-09-11 08:25:39.144152143 +0000 UTC m=+0.135721682 container create 932633c9d73ca13d27e7d7baa87b5035b7c3950aeaa57c0aa456ff887a5b7fa2 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, release=239.1651231664, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.openshift.expose-services=, distribution-scope=public) 2024-09-11T08:25:39.379 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 podman[63026]: 2024-09-11 08:25:39.379629453 +0000 UTC m=+0.371198941 container init 932633c9d73ca13d27e7d7baa87b5035b7c3950aeaa57c0aa456ff887a5b7fa2 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a, distribution-scope=public, version=8.5, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, name=ubi8, maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664) 2024-09-11T08:25:39.408 INFO:teuthology.orchestra.run.smithi110.stdout:210453397525 2024-09-11T08:25:39.630 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-mon[32601]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:39.630 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2162579774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:25:39.630 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 podman[63026]: 2024-09-11 08:25:39.382643909 +0000 UTC m=+0.374213392 container start 932633c9d73ca13d27e7d7baa87b5035b7c3950aeaa57c0aa456ff887a5b7fa2 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a, io.buildah.version=1.32.0, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, release=239.1651231664, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.openshift.expose-services=, version=8.5) 2024-09-11T08:25:39.630 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 bash[63026]: 932633c9d73ca13d27e7d7baa87b5035b7c3950aeaa57c0aa456ff887a5b7fa2 2024-09-11T08:25:39.630 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 systemd[1]: Started Ceph grafana.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:25:39.630 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.546929837Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547135153Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547154033Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547165562Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547173079Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547179344Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547185107Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.54719091Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547196978Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-11T08:25:39.631 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.54720287Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-11T08:25:39.632 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.54720855Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-11T08:25:39.632 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547214183Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-11T08:25:39.632 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.54722007Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-11T08:25:39.632 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=settings t=2024-09-11T08:25:39.547225963Z level=info msg="App mode production" 2024-09-11T08:25:39.632 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=sqlstore t=2024-09-11T08:25:39.547274768Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-11T08:25:39.632 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=sqlstore t=2024-09-11T08:25:39.547290287Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-11T08:25:39.632 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.548114301Z level=info msg="Starting DB migrations" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.548185778Z level=info msg="Executing migration" id="create migration_log table" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.548729113Z level=info msg="Executing migration" id="create user table" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.549149773Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.55105937Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.551520176Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.553507423Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.555356071Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.557218501Z level=info msg="Executing migration" id="create user table v2" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.561046791Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.561469581Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.565317356Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-11T08:25:39.633 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.569093699Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.57298844Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.573406545Z level=info msg="Executing migration" id="Update user table charset" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.577299423Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.581049593Z level=info msg="Executing migration" id="Add missing user data" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.581537473Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.585340318Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.589347971Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.593161696Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-11T08:25:39.634 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.596918938Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.601039753Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.604733424Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.60527687Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.60920847Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.612906441Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.614769229Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.616669148Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.618509618Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.618890288Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.620729953Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.621404384Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.62268759Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-11T08:25:39.635 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.624428317Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-11T08:25:39.636 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.624840553Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-11T08:25:39.636 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.626733956Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-11T08:25:39.636 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.627127481Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-11T08:25:39.636 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.628603125Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-11T08:25:39.636 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.628944296Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-11T08:25:39.636 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.630466046Z level=info msg="Executing migration" id="create star table" 2024-09-11T08:25:39.636 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.630920365Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-11T08:25:39.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:39 smithi110 ceph-mon[39404]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:39.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:39 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2162579774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:25:39.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:39 smithi110 ceph-mon[28998]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:39.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:39 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2162579774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-11T08:25:39.881 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.632269816Z level=info msg="Executing migration" id="create org table v1" 2024-09-11T08:25:39.881 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.634097662Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-11T08:25:39.881 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.635997586Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-11T08:25:39.881 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.637884721Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-11T08:25:39.881 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.63979226Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.641716587Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.643655805Z level=info msg="Executing migration" id="Update org table charset" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.645533943Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.647431501Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.649265602Z level=info msg="Executing migration" id="create dashboard table" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.651150492Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.652910599Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.654821851Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.656703954Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.658718197Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.660429492Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-11T08:25:39.882 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.6625196Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-11T08:25:39.883 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.664325561Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-11T08:25:39.883 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.666121908Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-11T08:25:39.883 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.670069416Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.6737603Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.67772295Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.681596241Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.682093408Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.683997572Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.684656208Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.685806502Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.687627869Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.688502604Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.689430125Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.690207761Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-11T08:25:39.884 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.691085629Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.691860276Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.692372105Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.693245837Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.694103027Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.695066042Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.695807752Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.696679506Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.697660047Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.698420779Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.700435733Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.70150301Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-11T08:25:39.885 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.702495487Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.704217241Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.705101948Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.707949713Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.710800425Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.713554536Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.716262671Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.718835009Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.721530211Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.724362913Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.727034498Z level=info msg="Executing migration" id="create data_source table" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.729842403Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.732458702Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-11T08:25:39.886 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.73314476Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.735733363Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.736175829Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.73887854Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.741487029Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.744135583Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.746820995Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.75117787Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.753886334Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.756420867Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.759207793Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.761888351Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.764371348Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-11T08:25:39.887 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.766844139Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.769623906Z level=info msg="Executing migration" id="Add uid column" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.772313267Z level=info msg="Executing migration" id="Update uid value" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.775015518Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.777577925Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.780340482Z level=info msg="Executing migration" id="create api_key table" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.782927631Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.785593438Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.788178831Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.790639729Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.79332295Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.795924603Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.796540992Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-11T08:25:39.888 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.797759411Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.798542576Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.798999155Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.799453359Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.79992301Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.80024548Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.800643771Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.801029203Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.801667141Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.802259762Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.802616187Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.803236837Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.803849489Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-11T08:25:39.889 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.804286393Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.804689414Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.805146083Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.805579814Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.806012617Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.806484648Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.806802274Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.807087727Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.807721427Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.808325544Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.808666497Z level=info msg="Executing migration" id="create quota table v1" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.809121295Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.80960115Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-11T08:25:39.890 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.809916276Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-11T08:25:39.891 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.810366036Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-11T08:25:39.891 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.810833423Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-11T08:25:39.891 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.811450404Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-11T08:25:39.891 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.81177501Z level=info msg="Executing migration" id="create session table" 2024-09-11T08:25:39.891 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.81223022Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-11T08:25:39.891 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.812521502Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.812852413Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.813280799Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.813734736Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.81403738Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.814319429Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.814649522Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.814971101Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.815400877Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-11T08:25:39.892 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.815722032Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.816311546Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.81665339Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.817302598Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.817984446Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.818306457Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.818774259Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.819247888Z level=info msg="Executing migration" id="create alert table v1" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.819722614Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.820606126Z level=info msg="Executing migration" id="add index alert state" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.821060825Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.821513139Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.821933276Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-11T08:25:39.893 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.822386188Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.822828615Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.824319774Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.824754851Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.825214927Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.825544221Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.825947721Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.826320036Z level=info msg="Executing migration" id="Add column is_default" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.827010113Z level=info msg="Executing migration" id="Add column frequency" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.827678685Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.82833735Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.829042877Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.829518701Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.829830284Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-11T08:25:39.894 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.830113865Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.830517578Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.830977626Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.831348867Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.831798747Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.832263177Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.833034724Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.833928608Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.834523452Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.835309031Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.83619833Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.83715421Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.837509684Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-11T08:25:39.895 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.840367939Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.84307247Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.845846224Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.846357879Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.84897662Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.851746317Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.854321914Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.854887472Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.857659339Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.860079828Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.862668174Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.863215561Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.865887313Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-11T08:25:39.896 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.868643414Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.86953434Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.870087661Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.871714905Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.872699435Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.873362383Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.873718317Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.874683859Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.875351835Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.876184593Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.876948276Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.877836485Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.878521772Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-11T08:25:39.897 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.879496132Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.880361079Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.881273401Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.881920388Z level=info msg="Executing migration" id="Move region to single row" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.88274135Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.883156203Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.88397163Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.884422791Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.884860358Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.885683818Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.886117119Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-11T08:25:39.898 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.886422767Z level=info msg="Executing migration" id="create test_data table" 2024-09-11T08:25:40.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397522 got 210453397525 for osd.7 2024-09-11T08:25:40.079 DEBUG:teuthology.parallel:result is None 2024-09-11T08:25:40.079 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-11T08:25:40.079 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:25:40.133 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.886852065Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-11T08:25:40.133 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.887277042Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-11T08:25:40.133 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.888121846Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-11T08:25:40.133 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.88896751Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.889303495Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.890169885Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.890992428Z level=info msg="Executing migration" id="create team table" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.891839208Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.892297784Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.893114472Z level=info msg="Executing migration" id="create team member table" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.893963409Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.896903356Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.897361473Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.900204658Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.902826782Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-11T08:25:40.134 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.905569474Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-11T08:25:40.135 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.906283156Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-11T08:25:40.135 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.909204379Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-11T08:25:40.135 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.911887527Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-11T08:25:40.136 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.914580779Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-11T08:25:40.136 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.915026275Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-11T08:25:40.136 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.917773056Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-11T08:25:40.136 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.920402627Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-11T08:25:40.136 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.923368149Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.926150648Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.929009319Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.931585918Z level=info msg="Executing migration" id="create tag table" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.934518602Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.937315215Z level=info msg="Executing migration" id="create login attempt table" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.939815365Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.942574477Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.945269857Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.948089477Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.95057346Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.953289829Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.955877188Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-11T08:25:40.137 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.958382505Z level=info msg="Executing migration" id="create user auth table" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.961110406Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.96375651Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.966261084Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.968953415Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.971788891Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.974435636Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.976968746Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.979728204Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.982401556Z level=info msg="Executing migration" id="create server_lock table" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.983203774Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.984234288Z level=info msg="Executing migration" id="create user auth token table" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.984865225Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-11T08:25:40.138 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.985692284Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.986575705Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.987422918Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.98821813Z level=info msg="Executing migration" id="create cache_data table" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.98904056Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.990000576Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.990628258Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.991575015Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.992276412Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.993085097Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.993565377Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.99401632Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-11T08:25:40.139 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.994829534Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.995679186Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.996545838Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.997551958Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.998407628Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:39.99914127Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.00002249Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.00084566Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.001827231Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.002643742Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.0033561Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.00432214Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.005033556Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-11T08:25:40.140 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.005983484Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.006519996Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.006969376Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.007808356Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.008660535Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.009555504Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.010495342Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.013905286Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.017317511Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.020080679Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.022810712Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.025606327Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.028182374Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-11T08:25:40.141 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.03091796Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.033651354Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.036548162Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.039138319Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.040023627Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.04101807Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.04210527Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.042892626Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.043869421Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.044846701Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.045683204Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.046578838Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.047403881Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-11T08:25:40.142 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.048260462Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.049201614Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.04973244Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.050171545Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.050967082Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.051292033Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.052167846Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.053054506Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.053940018Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.054792606Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.0556474Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.056502762Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.057488092Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-11T08:25:40.143 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.058384287Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.059350399Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.062083545Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.064971309Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.067584641Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.070527719Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.07314484Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.075937069Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.078825838Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.081593032Z level=info msg="Executing migration" id="create alert_image table" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.084279505Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.087297933Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.089983335Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-11T08:25:40.144 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.092778663Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.093261804Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.093346441Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.095981339Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.098842468Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.101574668Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.10183598Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.10196305Z level=info msg="alerts found to migrate" alerts=0 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.104258099Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.107067893Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.109811071Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.11242464Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-11T08:25:40.145 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.115173738Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-11T08:25:40.146 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.117948308Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-11T08:25:40.146 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.120601657Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-11T08:25:40.146 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.123195928Z level=info msg="Executing migration" id="create data_keys table" 2024-09-11T08:25:40.146 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.125788649Z level=info msg="Executing migration" id="create secrets table" 2024-09-11T08:25:40.146 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.128565472Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-11T08:25:40.146 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.132911454Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-11T08:25:40.146 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.135697992Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-11T08:25:40.370 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:40.384 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.13843689Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-11T08:25:40.384 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.14281667Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-11T08:25:40.384 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.14719728Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-11T08:25:40.384 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.149754858Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.152433601Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.155064001Z level=info msg="Executing migration" id="create permission table" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.155722367Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.158543643Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.161172759Z level=info msg="Executing migration" id="create role table" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.161859678Z level=info msg="Executing migration" id="add column display_name" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.162862296Z level=info msg="Executing migration" id="add column group_name" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.163830604Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.164679609Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.165629792Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.166276529Z level=info msg="Executing migration" id="create team role table" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.167227094Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-11T08:25:40.385 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.167886989Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.168696112Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.169454284Z level=info msg="Executing migration" id="create user role table" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.170370169Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.171066458Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.171924047Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.172746583Z level=info msg="Executing migration" id="create builtin role table" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.173617794Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.174514721Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.175257511Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.176312096Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.17716065Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.177733508Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-11T08:25:40.386 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.178158778Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.178559486Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.178958719Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.179391794Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.180364112Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.181175738Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.18164726Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.181944739Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-11T08:25:40.387 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.182372654Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-11T08:25:40.388 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.182856546Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-11T08:25:40.388 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.18326172Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-11T08:25:40.388 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.183554721Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-11T08:25:40.388 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.183945159Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-11T08:25:40.388 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.184365105Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-11T08:25:40.388 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.184839367Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-11T08:25:40.388 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.185836505Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.186165584Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.186603295Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.187037252Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.187502415Z level=info msg="Executing migration" id="add correlation config column" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.188534217Z level=info msg="Executing migration" id="create entity_events table" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.189141543Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.189574325Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.189691078Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.189987807Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.190066403Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.190308221Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-11T08:25:40.389 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.190706998Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.19117337Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.192063758Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.192909447Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.193739149Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.19473938Z level=info msg="Executing migration" id="Drop public config table" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.195536479Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.196441757Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.197185994Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.19817678Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.198917762Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.20200085Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.203045043Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-11T08:25:40.390 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.20585925Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.208680908Z level=info msg="Executing migration" id="add share column" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.211328051Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.214221256Z level=info msg="Executing migration" id="create file table" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.216937097Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.219584159Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.222370668Z level=info msg="Executing migration" id="create file_meta table" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.222814747Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.225672037Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.228375804Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.231209719Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.233969798Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-11T08:25:40.391 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.236567563Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.239264456Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.240406569Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.243297899Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.246079072Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.248792098Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.251495489Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.254312913Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.256991575Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.259883159Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.269800351Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.272666494Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-11T08:25:40.392 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.275559613Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.279000331Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.281794685Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.284755607Z level=info msg="Executing migration" id="create folder table" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.287455118Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.290062563Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.292720792Z level=info msg="Executing migration" id="Update folder title length" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.295294853Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=migrator t=2024-09-11T08:25:40.297987011Z level=info msg="migrations completed" performed=485 skipped=0 duration=749.822248ms 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=sqlstore t=2024-09-11T08:25:40.298376321Z level=info msg="Created default organization" 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=plugin.loader t=2024-09-11T08:25:40.355133155Z level=info msg="Plugin registered" pluginID=input 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=plugin.loader t=2024-09-11T08:25:40.382454081Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=plugin.loader t=2024-09-11T08:25:40.382549182Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-11T08:25:40.393 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=secrets t=2024-09-11T08:25:40.382693273Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-11T08:25:40.394 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=query_data t=2024-09-11T08:25:40.384413752Z level=info msg="Query Service initialization" 2024-09-11T08:25:40.768 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.768 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.768 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.768 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.768 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[28998]: Deploying daemon node-exporter.a on smithi110 2024-09-11T08:25:40.768 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:40.768 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[39404]: Deploying daemon node-exporter.a on smithi110 2024-09-11T08:25:40.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:40.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.820 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=live.push_http t=2024-09-11T08:25:40.387616451Z level=info msg="Live Push Gateway initialization" 2024-09-11T08:25:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-mon[32601]: Deploying daemon node-exporter.a on smithi110 2024-09-11T08:25:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:41.560 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=infra.usagestats.collector t=2024-09-11T08:25:41.141255584Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-11T08:25:41.560 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=provisioning.datasources t=2024-09-11T08:25:41.141725951Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-11T08:25:41.560 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=provisioning.datasources t=2024-09-11T08:25:41.174000646Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-11T08:25:41.560 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=provisioning.datasources t=2024-09-11T08:25:41.205698287Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-11T08:25:41.560 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=provisioning.alerting t=2024-09-11T08:25:41.22718962Z level=info msg="starting to provision alerting" 2024-09-11T08:25:41.560 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=provisioning.alerting t=2024-09-11T08:25:41.227234331Z level=info msg="finished to provision alerting" 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=grafanaStorageLogger t=2024-09-11T08:25:41.227368953Z level=info msg="storage starting" 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=ngalert.state.manager t=2024-09-11T08:25:41.227396292Z level=info msg="Warming state cache for startup" 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=http.server t=2024-09-11T08:25:41.229091375Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=sqlstore.transactions t=2024-09-11T08:25:41.237935084Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=ngalert.state.manager t=2024-09-11T08:25:41.241968809Z level=info msg="State cache has been initialized" states=0 duration=14.571859ms 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=ticker t=2024-09-11T08:25:41.242020254Z level=info msg=starting first_tick=2024-09-11T08:25:50Z 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=ngalert.multiorg.alertmanager t=2024-09-11T08:25:41.242033067Z level=info msg="starting MultiOrg Alertmanager" 2024-09-11T08:25:41.561 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a[63040]: logger=sqlstore.transactions t=2024-09-11T08:25:41.278310274Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-11T08:25:41.798 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:41 smithi110 ceph-mon[28998]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:41.798 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:41 smithi110 ceph-mon[39404]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:41.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:41 smithi139 ceph-mon[32601]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:42.218 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:25:42.220 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-11T08:25:42.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:42 smithi139 ceph-mon[32601]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:25:42.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:42 smithi110 ceph-mon[28998]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:25:42.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:42 smithi110 ceph-mon[39404]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:25:43.632 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-11T08:25:40.977794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":926,"num_read_kb":783,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":548,"ondisk_log_size":548,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385448,"kb_used_data":6604,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575768,"statfs":{"total":767960285184,"available":765517586432,"internally_reserved":0,"allocated":6762496,"data_stored":3649696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002493"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306862+0000","last_change":"2024-09-11T08:24:25.338389+0000","last_active":"2024-09-11T08:24:31.306862+0000","last_peered":"2024-09-11T08:24:31.306862+0000","last_clean":"2024-09-11T08:24:31.306862+0000","last_became_active":"2024-09-11T08:24:25.337868+0000","last_became_peered":"2024-09-11T08:24:25.337868+0000","last_unstale":"2024-09-11T08:24:31.306862+0000","last_undegraded":"2024-09-11T08:24:31.306862+0000","last_fullsized":"2024-09-11T08:24:31.306862+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:27:32.651542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121531+0000","last_change":"2024-09-11T08:24:18.607982+0000","last_active":"2024-09-11T08:24:31.121531+0000","last_peered":"2024-09-11T08:24:31.121531+0000","last_clean":"2024-09-11T08:24:31.121531+0000","last_became_active":"2024-09-11T08:24:18.607693+0000","last_became_peered":"2024-09-11T08:24:18.607693+0000","last_unstale":"2024-09-11T08:24:31.121531+0000","last_undegraded":"2024-09-11T08:24:31.121531+0000","last_fullsized":"2024-09-11T08:24:31.121531+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:20:37.741409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306917+0000","last_change":"2024-09-11T08:24:20.616820+0000","last_active":"2024-09-11T08:24:31.306917+0000","last_peered":"2024-09-11T08:24:31.306917+0000","last_clean":"2024-09-11T08:24:31.306917+0000","last_became_active":"2024-09-11T08:24:20.616359+0000","last_became_peered":"2024-09-11T08:24:20.616359+0000","last_unstale":"2024-09-11T08:24:31.306917+0000","last_undegraded":"2024-09-11T08:24:31.306917+0000","last_fullsized":"2024-09-11T08:24:31.306917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:43.193298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120987+0000","last_change":"2024-09-11T08:24:22.622622+0000","last_active":"2024-09-11T08:24:31.120987+0000","last_peered":"2024-09-11T08:24:31.120987+0000","last_clean":"2024-09-11T08:24:31.120987+0000","last_became_active":"2024-09-11T08:24:22.622225+0000","last_became_peered":"2024-09-11T08:24:22.622225+0000","last_unstale":"2024-09-11T08:24:31.120987+0000","last_undegraded":"2024-09-11T08:24:31.120987+0000","last_fullsized":"2024-09-11T08:24:31.120987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:56.926961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307464+0000","last_change":"2024-09-11T08:24:18.606375+0000","last_active":"2024-09-11T08:24:31.307464+0000","last_peered":"2024-09-11T08:24:31.307464+0000","last_clean":"2024-09-11T08:24:31.307464+0000","last_became_active":"2024-09-11T08:24:18.606112+0000","last_became_peered":"2024-09-11T08:24:18.606112+0000","last_unstale":"2024-09-11T08:24:31.307464+0000","last_undegraded":"2024-09-11T08:24:31.307464+0000","last_fullsized":"2024-09-11T08:24:31.307464+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:40:12.874596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120996+0000","last_change":"2024-09-11T08:24:20.608184+0000","last_active":"2024-09-11T08:24:31.120996+0000","last_peered":"2024-09-11T08:24:31.120996+0000","last_clean":"2024-09-11T08:24:31.120996+0000","last_became_active":"2024-09-11T08:24:20.607942+0000","last_became_peered":"2024-09-11T08:24:20.607942+0000","last_unstale":"2024-09-11T08:24:31.120996+0000","last_undegraded":"2024-09-11T08:24:31.120996+0000","last_fullsized":"2024-09-11T08:24:31.120996+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.976179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643685+0000","last_change":"2024-09-11T08:24:22.621538+0000","last_active":"2024-09-11T08:24:31.643685+0000","last_peered":"2024-09-11T08:24:31.643685+0000","last_clean":"2024-09-11T08:24:31.643685+0000","last_became_active":"2024-09-11T08:24:22.621254+0000","last_became_peered":"2024-09-11T08:24:22.621254+0000","last_unstale":"2024-09-11T08:24:31.643685+0000","last_undegraded":"2024-09-11T08:24:31.643685+0000","last_fullsized":"2024-09-11T08:24:31.643685+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:11:18.928367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121314+0000","last_change":"2024-09-11T08:24:24.630199+0000","last_active":"2024-09-11T08:24:31.121314+0000","last_peered":"2024-09-11T08:24:31.121314+0000","last_clean":"2024-09-11T08:24:31.121314+0000","last_became_active":"2024-09-11T08:24:24.629091+0000","last_became_peered":"2024-09-11T08:24:24.629091+0000","last_unstale":"2024-09-11T08:24:31.121314+0000","last_undegraded":"2024-09-11T08:24:31.121314+0000","last_fullsized":"2024-09-11T08:24:31.121314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:01:56.296158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645560+0000","last_change":"2024-09-11T08:24:18.612232+0000","last_active":"2024-09-11T08:24:31.645560+0000","last_peered":"2024-09-11T08:24:31.645560+0000","last_clean":"2024-09-11T08:24:31.645560+0000","last_became_active":"2024-09-11T08:24:18.607618+0000","last_became_peered":"2024-09-11T08:24:18.607618+0000","last_unstale":"2024-09-11T08:24:31.645560+0000","last_undegraded":"2024-09-11T08:24:31.645560+0000","last_fullsized":"2024-09-11T08:24:31.645560+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:02.476467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"60'5","reported_seq":26,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121198+0000","last_change":"2024-09-11T08:24:20.614170+0000","last_active":"2024-09-11T08:24:31.121198+0000","last_peered":"2024-09-11T08:24:31.121198+0000","last_clean":"2024-09-11T08:24:31.121198+0000","last_became_active":"2024-09-11T08:24:20.613851+0000","last_became_peered":"2024-09-11T08:24:20.613851+0000","last_unstale":"2024-09-11T08:24:31.121198+0000","last_undegraded":"2024-09-11T08:24:31.121198+0000","last_fullsized":"2024-09-11T08:24:31.121198+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:43.175051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644181+0000","last_change":"2024-09-11T08:24:22.628941+0000","last_active":"2024-09-11T08:24:31.644181+0000","last_peered":"2024-09-11T08:24:31.644181+0000","last_clean":"2024-09-11T08:24:31.644181+0000","last_became_active":"2024-09-11T08:24:22.628649+0000","last_became_peered":"2024-09-11T08:24:22.628649+0000","last_unstale":"2024-09-11T08:24:31.644181+0000","last_undegraded":"2024-09-11T08:24:31.644181+0000","last_fullsized":"2024-09-11T08:24:31.644181+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:50:19.270029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643770+0000","last_change":"2024-09-11T08:24:24.636731+0000","last_active":"2024-09-11T08:24:31.643770+0000","last_peered":"2024-09-11T08:24:31.643770+0000","last_clean":"2024-09-11T08:24:31.643770+0000","last_became_active":"2024-09-11T08:24:24.636384+0000","last_became_peered":"2024-09-11T08:24:24.636384+0000","last_unstale":"2024-09-11T08:24:31.643770+0000","last_undegraded":"2024-09-11T08:24:31.643770+0000","last_fullsized":"2024-09-11T08:24:31.643770+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:46.349874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645295+0000","last_change":"2024-09-11T08:24:18.611777+0000","last_active":"2024-09-11T08:24:31.645295+0000","last_peered":"2024-09-11T08:24:31.645295+0000","last_clean":"2024-09-11T08:24:31.645295+0000","last_became_active":"2024-09-11T08:24:18.611494+0000","last_became_peered":"2024-09-11T08:24:18.611494+0000","last_unstale":"2024-09-11T08:24:31.645295+0000","last_undegraded":"2024-09-11T08:24:31.645295+0000","last_fullsized":"2024-09-11T08:24:31.645295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:24:14.345542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121393+0000","last_change":"2024-09-11T08:24:20.613540+0000","last_active":"2024-09-11T08:24:31.121393+0000","last_peered":"2024-09-11T08:24:31.121393+0000","last_clean":"2024-09-11T08:24:31.121393+0000","last_became_active":"2024-09-11T08:24:20.613272+0000","last_became_peered":"2024-09-11T08:24:20.613272+0000","last_unstale":"2024-09-11T08:24:31.121393+0000","last_undegraded":"2024-09-11T08:24:31.121393+0000","last_fullsized":"2024-09-11T08:24:31.121393+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:03.597166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644161+0000","last_change":"2024-09-11T08:24:22.624351+0000","last_active":"2024-09-11T08:24:31.644161+0000","last_peered":"2024-09-11T08:24:31.644161+0000","last_clean":"2024-09-11T08:24:31.644161+0000","last_became_active":"2024-09-11T08:24:22.623880+0000","last_became_peered":"2024-09-11T08:24:22.623880+0000","last_unstale":"2024-09-11T08:24:31.644161+0000","last_undegraded":"2024-09-11T08:24:31.644161+0000","last_fullsized":"2024-09-11T08:24:31.644161+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:39.388477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121099+0000","last_change":"2024-09-11T08:24:24.631898+0000","last_active":"2024-09-11T08:24:31.121099+0000","last_peered":"2024-09-11T08:24:31.121099+0000","last_clean":"2024-09-11T08:24:31.121099+0000","last_became_active":"2024-09-11T08:24:24.631310+0000","last_became_peered":"2024-09-11T08:24:24.631310+0000","last_unstale":"2024-09-11T08:24:31.121099+0000","last_undegraded":"2024-09-11T08:24:31.121099+0000","last_fullsized":"2024-09-11T08:24:31.121099+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:27.906608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"60'17","reported_seq":44,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121646+0000","last_change":"2024-09-11T08:24:20.606737+0000","last_active":"2024-09-11T08:24:31.121646+0000","last_peered":"2024-09-11T08:24:31.121646+0000","last_clean":"2024-09-11T08:24:31.121646+0000","last_became_active":"2024-09-11T08:24:20.606386+0000","last_became_peered":"2024-09-11T08:24:20.606386+0000","last_unstale":"2024-09-11T08:24:31.121646+0000","last_undegraded":"2024-09-11T08:24:31.121646+0000","last_fullsized":"2024-09-11T08:24:31.121646+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:28.536648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644791+0000","last_change":"2024-09-11T08:24:18.612097+0000","last_active":"2024-09-11T08:24:31.644791+0000","last_peered":"2024-09-11T08:24:31.644791+0000","last_clean":"2024-09-11T08:24:31.644791+0000","last_became_active":"2024-09-11T08:24:18.611834+0000","last_became_peered":"2024-09-11T08:24:18.611834+0000","last_unstale":"2024-09-11T08:24:31.644791+0000","last_undegraded":"2024-09-11T08:24:31.644791+0000","last_fullsized":"2024-09-11T08:24:31.644791+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:52:12.575391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643836+0000","last_change":"2024-09-11T08:24:22.628019+0000","last_active":"2024-09-11T08:24:31.643836+0000","last_peered":"2024-09-11T08:24:31.643836+0000","last_clean":"2024-09-11T08:24:31.643836+0000","last_became_active":"2024-09-11T08:24:22.627731+0000","last_became_peered":"2024-09-11T08:24:22.627731+0000","last_unstale":"2024-09-11T08:24:31.643836+0000","last_undegraded":"2024-09-11T08:24:31.643836+0000","last_fullsized":"2024-09-11T08:24:31.643836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:59:30.448604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121324+0000","last_change":"2024-09-11T08:24:24.636831+0000","last_active":"2024-09-11T08:24:31.121324+0000","last_peered":"2024-09-11T08:24:31.121324+0000","last_clean":"2024-09-11T08:24:31.121324+0000","last_became_active":"2024-09-11T08:24:24.636483+0000","last_became_peered":"2024-09-11T08:24:24.636483+0000","last_unstale":"2024-09-11T08:24:31.121324+0000","last_undegraded":"2024-09-11T08:24:31.121324+0000","last_fullsized":"2024-09-11T08:24:31.121324+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:06:41.420567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121422+0000","last_change":"2024-09-11T08:24:20.613625+0000","last_active":"2024-09-11T08:24:31.121422+0000","last_peered":"2024-09-11T08:24:31.121422+0000","last_clean":"2024-09-11T08:24:31.121422+0000","last_became_active":"2024-09-11T08:24:20.613354+0000","last_became_peered":"2024-09-11T08:24:20.613354+0000","last_unstale":"2024-09-11T08:24:31.121422+0000","last_undegraded":"2024-09-11T08:24:31.121422+0000","last_fullsized":"2024-09-11T08:24:31.121422+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:59.818031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"53'1","reported_seq":25,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644036+0000","last_change":"2024-09-11T08:24:18.607601+0000","last_active":"2024-09-11T08:24:31.644036+0000","last_peered":"2024-09-11T08:24:31.644036+0000","last_clean":"2024-09-11T08:24:31.644036+0000","last_became_active":"2024-09-11T08:24:18.607085+0000","last_became_peered":"2024-09-11T08:24:18.607085+0000","last_unstale":"2024-09-11T08:24:31.644036+0000","last_undegraded":"2024-09-11T08:24:31.644036+0000","last_fullsized":"2024-09-11T08:24:31.644036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:12:35.685598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690265+0000","last_change":"2024-09-11T08:24:22.619694+0000","last_active":"2024-09-11T08:25:31.690265+0000","last_peered":"2024-09-11T08:25:31.690265+0000","last_clean":"2024-09-11T08:25:31.690265+0000","last_became_active":"2024-09-11T08:24:22.619452+0000","last_became_peered":"2024-09-11T08:24:22.619452+0000","last_unstale":"2024-09-11T08:25:31.690265+0000","last_undegraded":"2024-09-11T08:25:31.690265+0000","last_fullsized":"2024-09-11T08:25:31.690265+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:55:57.467687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121552+0000","last_change":"2024-09-11T08:24:24.634484+0000","last_active":"2024-09-11T08:24:31.121552+0000","last_peered":"2024-09-11T08:24:31.121552+0000","last_clean":"2024-09-11T08:24:31.121552+0000","last_became_active":"2024-09-11T08:24:24.634134+0000","last_became_peered":"2024-09-11T08:24:24.634134+0000","last_unstale":"2024-09-11T08:24:31.121552+0000","last_undegraded":"2024-09-11T08:24:31.121552+0000","last_fullsized":"2024-09-11T08:24:31.121552+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:36:18.363418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644631+0000","last_change":"2024-09-11T08:24:20.610686+0000","last_active":"2024-09-11T08:24:31.644631+0000","last_peered":"2024-09-11T08:24:31.644631+0000","last_clean":"2024-09-11T08:24:31.644631+0000","last_became_active":"2024-09-11T08:24:20.610337+0000","last_became_peered":"2024-09-11T08:24:20.610337+0000","last_unstale":"2024-09-11T08:24:31.644631+0000","last_undegraded":"2024-09-11T08:24:31.644631+0000","last_fullsized":"2024-09-11T08:24:31.644631+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:07.158347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643904+0000","last_change":"2024-09-11T08:24:18.606710+0000","last_active":"2024-09-11T08:24:31.643904+0000","last_peered":"2024-09-11T08:24:31.643904+0000","last_clean":"2024-09-11T08:24:31.643904+0000","last_became_active":"2024-09-11T08:24:18.606432+0000","last_became_peered":"2024-09-11T08:24:18.606432+0000","last_unstale":"2024-09-11T08:24:31.643904+0000","last_undegraded":"2024-09-11T08:24:31.643904+0000","last_fullsized":"2024-09-11T08:24:31.643904+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:18.778917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690336+0000","last_change":"2024-09-11T08:24:22.624344+0000","last_active":"2024-09-11T08:25:31.690336+0000","last_peered":"2024-09-11T08:25:31.690336+0000","last_clean":"2024-09-11T08:25:31.690336+0000","last_became_active":"2024-09-11T08:24:22.623989+0000","last_became_peered":"2024-09-11T08:24:22.623989+0000","last_unstale":"2024-09-11T08:25:31.690336+0000","last_undegraded":"2024-09-11T08:25:31.690336+0000","last_fullsized":"2024-09-11T08:25:31.690336+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:15:36.367906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644890+0000","last_change":"2024-09-11T08:24:24.633791+0000","last_active":"2024-09-11T08:24:31.644890+0000","last_peered":"2024-09-11T08:24:31.644890+0000","last_clean":"2024-09-11T08:24:31.644890+0000","last_became_active":"2024-09-11T08:24:24.633508+0000","last_became_peered":"2024-09-11T08:24:24.633508+0000","last_unstale":"2024-09-11T08:24:31.644890+0000","last_undegraded":"2024-09-11T08:24:31.644890+0000","last_fullsized":"2024-09-11T08:24:31.644890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:40:02.496883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121677+0000","last_change":"2024-09-11T08:24:20.608269+0000","last_active":"2024-09-11T08:24:31.121677+0000","last_peered":"2024-09-11T08:24:31.121677+0000","last_clean":"2024-09-11T08:24:31.121677+0000","last_became_active":"2024-09-11T08:24:20.607905+0000","last_became_peered":"2024-09-11T08:24:20.607905+0000","last_unstale":"2024-09-11T08:24:31.121677+0000","last_undegraded":"2024-09-11T08:24:31.121677+0000","last_fullsized":"2024-09-11T08:24:31.121677+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:37:51.123445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643733+0000","last_change":"2024-09-11T08:24:18.613184+0000","last_active":"2024-09-11T08:24:31.643733+0000","last_peered":"2024-09-11T08:24:31.643733+0000","last_clean":"2024-09-11T08:24:31.643733+0000","last_became_active":"2024-09-11T08:24:18.612354+0000","last_became_peered":"2024-09-11T08:24:18.612354+0000","last_unstale":"2024-09-11T08:24:31.643733+0000","last_undegraded":"2024-09-11T08:24:31.643733+0000","last_fullsized":"2024-09-11T08:24:31.643733+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:37:35.649285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645167+0000","last_change":"2024-09-11T08:24:22.625099+0000","last_active":"2024-09-11T08:24:31.645167+0000","last_peered":"2024-09-11T08:24:31.645167+0000","last_clean":"2024-09-11T08:24:31.645167+0000","last_became_active":"2024-09-11T08:24:22.623827+0000","last_became_peered":"2024-09-11T08:24:22.623827+0000","last_unstale":"2024-09-11T08:24:31.645167+0000","last_undegraded":"2024-09-11T08:24:31.645167+0000","last_fullsized":"2024-09-11T08:24:31.645167+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:46.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306926+0000","last_change":"2024-09-11T08:24:25.336583+0000","last_active":"2024-09-11T08:24:31.306926+0000","last_peered":"2024-09-11T08:24:31.306926+0000","last_clean":"2024-09-11T08:24:31.306926+0000","last_became_active":"2024-09-11T08:24:25.336233+0000","last_became_peered":"2024-09-11T08:24:25.336233+0000","last_unstale":"2024-09-11T08:24:31.306926+0000","last_undegraded":"2024-09-11T08:24:31.306926+0000","last_fullsized":"2024-09-11T08:24:31.306926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:12:21.502321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"60'14","reported_seq":37,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121135+0000","last_change":"2024-09-11T08:24:20.608854+0000","last_active":"2024-09-11T08:24:31.121135+0000","last_peered":"2024-09-11T08:24:31.121135+0000","last_clean":"2024-09-11T08:24:31.121135+0000","last_became_active":"2024-09-11T08:24:20.608582+0000","last_became_peered":"2024-09-11T08:24:20.608582+0000","last_unstale":"2024-09-11T08:24:31.121135+0000","last_undegraded":"2024-09-11T08:24:31.121135+0000","last_fullsized":"2024-09-11T08:24:31.121135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:13.064049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644543+0000","last_change":"2024-09-11T08:24:18.615828+0000","last_active":"2024-09-11T08:24:31.644543+0000","last_peered":"2024-09-11T08:24:31.644543+0000","last_clean":"2024-09-11T08:24:31.644543+0000","last_became_active":"2024-09-11T08:24:18.615393+0000","last_became_peered":"2024-09-11T08:24:18.615393+0000","last_unstale":"2024-09-11T08:24:31.644543+0000","last_undegraded":"2024-09-11T08:24:31.644543+0000","last_fullsized":"2024-09-11T08:24:31.644543+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:00.163725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690299+0000","last_change":"2024-09-11T08:24:22.630750+0000","last_active":"2024-09-11T08:25:31.690299+0000","last_peered":"2024-09-11T08:25:31.690299+0000","last_clean":"2024-09-11T08:25:31.690299+0000","last_became_active":"2024-09-11T08:24:22.629848+0000","last_became_peered":"2024-09-11T08:24:22.629848+0000","last_unstale":"2024-09-11T08:25:31.690299+0000","last_undegraded":"2024-09-11T08:25:31.690299+0000","last_fullsized":"2024-09-11T08:25:31.690299+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:21:46.662466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644610+0000","last_change":"2024-09-11T08:24:25.338123+0000","last_active":"2024-09-11T08:24:31.644610+0000","last_peered":"2024-09-11T08:24:31.644610+0000","last_clean":"2024-09-11T08:24:31.644610+0000","last_became_active":"2024-09-11T08:24:25.337812+0000","last_became_peered":"2024-09-11T08:24:25.337812+0000","last_unstale":"2024-09-11T08:24:31.644610+0000","last_undegraded":"2024-09-11T08:24:31.644610+0000","last_fullsized":"2024-09-11T08:24:31.644610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:50:28.210131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"60'18","reported_seq":48,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306788+0000","last_change":"2024-09-11T08:24:20.979676+0000","last_active":"2024-09-11T08:24:31.306788+0000","last_peered":"2024-09-11T08:24:31.306788+0000","last_clean":"2024-09-11T08:24:31.306788+0000","last_became_active":"2024-09-11T08:24:20.979423+0000","last_became_peered":"2024-09-11T08:24:20.979423+0000","last_unstale":"2024-09-11T08:24:31.306788+0000","last_undegraded":"2024-09-11T08:24:31.306788+0000","last_fullsized":"2024-09-11T08:24:31.306788+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:57:40.839035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644430+0000","last_change":"2024-09-11T08:24:18.609802+0000","last_active":"2024-09-11T08:24:31.644430+0000","last_peered":"2024-09-11T08:24:31.644430+0000","last_clean":"2024-09-11T08:24:31.644430+0000","last_became_active":"2024-09-11T08:24:18.609496+0000","last_became_peered":"2024-09-11T08:24:18.609496+0000","last_unstale":"2024-09-11T08:24:31.644430+0000","last_undegraded":"2024-09-11T08:24:31.644430+0000","last_fullsized":"2024-09-11T08:24:31.644430+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:38:29.491263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121059+0000","last_change":"2024-09-11T08:24:22.625367+0000","last_active":"2024-09-11T08:24:31.121059+0000","last_peered":"2024-09-11T08:24:31.121059+0000","last_clean":"2024-09-11T08:24:31.121059+0000","last_became_active":"2024-09-11T08:24:22.625102+0000","last_became_peered":"2024-09-11T08:24:22.625102+0000","last_unstale":"2024-09-11T08:24:31.121059+0000","last_undegraded":"2024-09-11T08:24:31.121059+0000","last_fullsized":"2024-09-11T08:24:31.121059+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:04:50.361263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121387+0000","last_change":"2024-09-11T08:24:24.634638+0000","last_active":"2024-09-11T08:24:31.121387+0000","last_peered":"2024-09-11T08:24:31.121387+0000","last_clean":"2024-09-11T08:24:31.121387+0000","last_became_active":"2024-09-11T08:24:24.634204+0000","last_became_peered":"2024-09-11T08:24:24.634204+0000","last_unstale":"2024-09-11T08:24:31.121387+0000","last_undegraded":"2024-09-11T08:24:31.121387+0000","last_fullsized":"2024-09-11T08:24:31.121387+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:10:24.991183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"60'19","reported_seq":52,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121860+0000","last_change":"2024-09-11T08:24:20.978326+0000","last_active":"2024-09-11T08:24:31.121860+0000","last_peered":"2024-09-11T08:24:31.121860+0000","last_clean":"2024-09-11T08:24:31.121860+0000","last_became_active":"2024-09-11T08:24:20.977927+0000","last_became_peered":"2024-09-11T08:24:20.977927+0000","last_unstale":"2024-09-11T08:24:31.121860+0000","last_undegraded":"2024-09-11T08:24:31.121860+0000","last_fullsized":"2024-09-11T08:24:31.121860+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:48.746531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645630+0000","last_change":"2024-09-11T08:24:18.606686+0000","last_active":"2024-09-11T08:24:31.645630+0000","last_peered":"2024-09-11T08:24:31.645630+0000","last_clean":"2024-09-11T08:24:31.645630+0000","last_became_active":"2024-09-11T08:24:18.606039+0000","last_became_peered":"2024-09-11T08:24:18.606039+0000","last_unstale":"2024-09-11T08:24:31.645630+0000","last_undegraded":"2024-09-11T08:24:31.645630+0000","last_fullsized":"2024-09-11T08:24:31.645630+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:18:54.869573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644952+0000","last_change":"2024-09-11T08:24:22.622052+0000","last_active":"2024-09-11T08:24:31.644952+0000","last_peered":"2024-09-11T08:24:31.644952+0000","last_clean":"2024-09-11T08:24:31.644952+0000","last_became_active":"2024-09-11T08:24:22.621771+0000","last_became_peered":"2024-09-11T08:24:22.621771+0000","last_unstale":"2024-09-11T08:24:31.644952+0000","last_undegraded":"2024-09-11T08:24:31.644952+0000","last_fullsized":"2024-09-11T08:24:31.644952+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:37.670351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643612+0000","last_change":"2024-09-11T08:24:25.336629+0000","last_active":"2024-09-11T08:24:31.643612+0000","last_peered":"2024-09-11T08:24:31.643612+0000","last_clean":"2024-09-11T08:24:31.643612+0000","last_became_active":"2024-09-11T08:24:25.336246+0000","last_became_peered":"2024-09-11T08:24:25.336246+0000","last_unstale":"2024-09-11T08:24:31.643612+0000","last_undegraded":"2024-09-11T08:24:31.643612+0000","last_fullsized":"2024-09-11T08:24:31.643612+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:50:23.195512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643957+0000","last_change":"2024-09-11T08:24:20.608306+0000","last_active":"2024-09-11T08:24:31.643957+0000","last_peered":"2024-09-11T08:24:31.643957+0000","last_clean":"2024-09-11T08:24:31.643957+0000","last_became_active":"2024-09-11T08:24:20.608045+0000","last_became_peered":"2024-09-11T08:24:20.608045+0000","last_unstale":"2024-09-11T08:24:31.643957+0000","last_undegraded":"2024-09-11T08:24:31.643957+0000","last_fullsized":"2024-09-11T08:24:31.643957+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:53:50.462569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644435+0000","last_change":"2024-09-11T08:24:18.607396+0000","last_active":"2024-09-11T08:24:31.644435+0000","last_peered":"2024-09-11T08:24:31.644435+0000","last_clean":"2024-09-11T08:24:31.644435+0000","last_became_active":"2024-09-11T08:24:18.607032+0000","last_became_peered":"2024-09-11T08:24:18.607032+0000","last_unstale":"2024-09-11T08:24:31.644435+0000","last_undegraded":"2024-09-11T08:24:31.644435+0000","last_fullsized":"2024-09-11T08:24:31.644435+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:00:43.052497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690269+0000","last_change":"2024-09-11T08:24:22.622109+0000","last_active":"2024-09-11T08:25:31.690269+0000","last_peered":"2024-09-11T08:25:31.690269+0000","last_clean":"2024-09-11T08:25:31.690269+0000","last_became_active":"2024-09-11T08:24:22.621836+0000","last_became_peered":"2024-09-11T08:24:22.621836+0000","last_unstale":"2024-09-11T08:25:31.690269+0000","last_undegraded":"2024-09-11T08:25:31.690269+0000","last_fullsized":"2024-09-11T08:25:31.690269+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:01:08.166600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120905+0000","last_change":"2024-09-11T08:24:24.634138+0000","last_active":"2024-09-11T08:24:31.120905+0000","last_peered":"2024-09-11T08:24:31.120905+0000","last_clean":"2024-09-11T08:24:31.120905+0000","last_became_active":"2024-09-11T08:24:24.633855+0000","last_became_peered":"2024-09-11T08:24:24.633855+0000","last_unstale":"2024-09-11T08:24:31.120905+0000","last_undegraded":"2024-09-11T08:24:31.120905+0000","last_fullsized":"2024-09-11T08:24:31.120905+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:59:36.676165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"60'28","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690277+0000","last_change":"2024-09-11T08:24:20.613279+0000","last_active":"2024-09-11T08:25:31.690277+0000","last_peered":"2024-09-11T08:25:31.690277+0000","last_clean":"2024-09-11T08:25:31.690277+0000","last_became_active":"2024-09-11T08:24:20.612856+0000","last_became_peered":"2024-09-11T08:24:20.612856+0000","last_unstale":"2024-09-11T08:25:31.690277+0000","last_undegraded":"2024-09-11T08:25:31.690277+0000","last_fullsized":"2024-09-11T08:25:31.690277+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:57.865486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"53'1","reported_seq":26,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644933+0000","last_change":"2024-09-11T08:24:18.613258+0000","last_active":"2024-09-11T08:24:31.644933+0000","last_peered":"2024-09-11T08:24:31.644933+0000","last_clean":"2024-09-11T08:24:31.644933+0000","last_became_active":"2024-09-11T08:24:18.612980+0000","last_became_peered":"2024-09-11T08:24:18.612980+0000","last_unstale":"2024-09-11T08:24:31.644933+0000","last_undegraded":"2024-09-11T08:24:31.644933+0000","last_fullsized":"2024-09-11T08:24:31.644933+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:56:37.282029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121545+0000","last_change":"2024-09-11T08:24:22.628056+0000","last_active":"2024-09-11T08:24:31.121545+0000","last_peered":"2024-09-11T08:24:31.121545+0000","last_clean":"2024-09-11T08:24:31.121545+0000","last_became_active":"2024-09-11T08:24:22.627650+0000","last_became_peered":"2024-09-11T08:24:22.627650+0000","last_unstale":"2024-09-11T08:24:31.121545+0000","last_undegraded":"2024-09-11T08:24:31.121545+0000","last_fullsized":"2024-09-11T08:24:31.121545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307249+0000","last_change":"2024-09-11T08:24:24.629826+0000","last_active":"2024-09-11T08:24:31.307249+0000","last_peered":"2024-09-11T08:24:31.307249+0000","last_clean":"2024-09-11T08:24:31.307249+0000","last_became_active":"2024-09-11T08:24:24.629549+0000","last_became_peered":"2024-09-11T08:24:24.629549+0000","last_unstale":"2024-09-11T08:24:31.307249+0000","last_undegraded":"2024-09-11T08:24:31.307249+0000","last_fullsized":"2024-09-11T08:24:31.307249+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:46:53.802170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"60'13","reported_seq":43,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644634+0000","last_change":"2024-09-11T08:24:20.610388+0000","last_active":"2024-09-11T08:24:31.644634+0000","last_peered":"2024-09-11T08:24:31.644634+0000","last_clean":"2024-09-11T08:24:31.644634+0000","last_became_active":"2024-09-11T08:24:20.610044+0000","last_became_peered":"2024-09-11T08:24:20.610044+0000","last_unstale":"2024-09-11T08:24:31.644634+0000","last_undegraded":"2024-09-11T08:24:31.644634+0000","last_fullsized":"2024-09-11T08:24:31.644634+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:43.169337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"60'5","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:39.027783+0000","last_change":"2024-09-11T08:24:22.614511+0000","last_active":"2024-09-11T08:25:39.027783+0000","last_peered":"2024-09-11T08:25:39.027783+0000","last_clean":"2024-09-11T08:25:39.027783+0000","last_became_active":"2024-09-11T08:24:18.608674+0000","last_became_peered":"2024-09-11T08:24:18.608674+0000","last_unstale":"2024-09-11T08:25:39.027783+0000","last_undegraded":"2024-09-11T08:25:39.027783+0000","last_fullsized":"2024-09-11T08:25:39.027783+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:32:47.672972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013623419999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120814+0000","last_change":"2024-09-11T08:24:18.604377+0000","last_active":"2024-09-11T08:24:31.120814+0000","last_peered":"2024-09-11T08:24:31.120814+0000","last_clean":"2024-09-11T08:24:31.120814+0000","last_became_active":"2024-09-11T08:24:18.604090+0000","last_became_peered":"2024-09-11T08:24:18.604090+0000","last_unstale":"2024-09-11T08:24:31.120814+0000","last_undegraded":"2024-09-11T08:24:31.120814+0000","last_fullsized":"2024-09-11T08:24:31.120814+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:03:20.969800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121503+0000","last_change":"2024-09-11T08:24:22.620313+0000","last_active":"2024-09-11T08:24:31.121503+0000","last_peered":"2024-09-11T08:24:31.121503+0000","last_clean":"2024-09-11T08:24:31.121503+0000","last_became_active":"2024-09-11T08:24:22.620041+0000","last_became_peered":"2024-09-11T08:24:22.620041+0000","last_unstale":"2024-09-11T08:24:31.121503+0000","last_undegraded":"2024-09-11T08:24:31.121503+0000","last_fullsized":"2024-09-11T08:24:31.121503+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:55:19.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645432+0000","last_change":"2024-09-11T08:24:25.339645+0000","last_active":"2024-09-11T08:24:31.645432+0000","last_peered":"2024-09-11T08:24:31.645432+0000","last_clean":"2024-09-11T08:24:31.645432+0000","last_became_active":"2024-09-11T08:24:25.339388+0000","last_became_peered":"2024-09-11T08:24:25.339388+0000","last_unstale":"2024-09-11T08:24:31.645432+0000","last_undegraded":"2024-09-11T08:24:31.645432+0000","last_fullsized":"2024-09-11T08:24:31.645432+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:14:58.317284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"60'12","reported_seq":34,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122146+0000","last_change":"2024-09-11T08:24:20.609894+0000","last_active":"2024-09-11T08:24:31.122146+0000","last_peered":"2024-09-11T08:24:31.122146+0000","last_clean":"2024-09-11T08:24:31.122146+0000","last_became_active":"2024-09-11T08:24:20.609623+0000","last_became_peered":"2024-09-11T08:24:20.609623+0000","last_unstale":"2024-09-11T08:24:31.122146+0000","last_undegraded":"2024-09-11T08:24:31.122146+0000","last_fullsized":"2024-09-11T08:24:31.122146+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:17:07.507564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"55'1","reported_seq":24,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121619+0000","last_change":"2024-09-11T08:24:22.619196+0000","last_active":"2024-09-11T08:24:31.121619+0000","last_peered":"2024-09-11T08:24:31.121619+0000","last_clean":"2024-09-11T08:24:31.121619+0000","last_became_active":"2024-09-11T08:24:18.606063+0000","last_became_peered":"2024-09-11T08:24:18.606063+0000","last_unstale":"2024-09-11T08:24:31.121619+0000","last_undegraded":"2024-09-11T08:24:31.121619+0000","last_fullsized":"2024-09-11T08:24:31.121619+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:01.586667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038483899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643797+0000","last_change":"2024-09-11T08:24:18.613336+0000","last_active":"2024-09-11T08:24:31.643797+0000","last_peered":"2024-09-11T08:24:31.643797+0000","last_clean":"2024-09-11T08:24:31.643797+0000","last_became_active":"2024-09-11T08:24:18.612757+0000","last_became_peered":"2024-09-11T08:24:18.612757+0000","last_unstale":"2024-09-11T08:24:31.643797+0000","last_undegraded":"2024-09-11T08:24:31.643797+0000","last_fullsized":"2024-09-11T08:24:31.643797+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:06:57.231216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645203+0000","last_change":"2024-09-11T08:24:22.626749+0000","last_active":"2024-09-11T08:24:31.645203+0000","last_peered":"2024-09-11T08:24:31.645203+0000","last_clean":"2024-09-11T08:24:31.645203+0000","last_became_active":"2024-09-11T08:24:22.626490+0000","last_became_peered":"2024-09-11T08:24:22.626490+0000","last_unstale":"2024-09-11T08:24:31.645203+0000","last_undegraded":"2024-09-11T08:24:31.645203+0000","last_fullsized":"2024-09-11T08:24:31.645203+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:24.400523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643993+0000","last_change":"2024-09-11T08:24:24.629842+0000","last_active":"2024-09-11T08:24:31.643993+0000","last_peered":"2024-09-11T08:24:31.643993+0000","last_clean":"2024-09-11T08:24:31.643993+0000","last_became_active":"2024-09-11T08:24:24.629559+0000","last_became_peered":"2024-09-11T08:24:24.629559+0000","last_unstale":"2024-09-11T08:24:31.643993+0000","last_undegraded":"2024-09-11T08:24:31.643993+0000","last_fullsized":"2024-09-11T08:24:31.643993+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:19:39.030608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"60'14","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644154+0000","last_change":"2024-09-11T08:24:20.980332+0000","last_active":"2024-09-11T08:24:31.644154+0000","last_peered":"2024-09-11T08:24:31.644154+0000","last_clean":"2024-09-11T08:24:31.644154+0000","last_became_active":"2024-09-11T08:24:20.979988+0000","last_became_peered":"2024-09-11T08:24:20.979988+0000","last_unstale":"2024-09-11T08:24:31.644154+0000","last_undegraded":"2024-09-11T08:24:31.644154+0000","last_fullsized":"2024-09-11T08:24:31.644154+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:20.859217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"57'2","reported_seq":25,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307095+0000","last_change":"2024-09-11T08:24:22.625005+0000","last_active":"2024-09-11T08:24:31.307095+0000","last_peered":"2024-09-11T08:24:31.307095+0000","last_clean":"2024-09-11T08:24:31.307095+0000","last_became_active":"2024-09-11T08:24:18.610583+0000","last_became_peered":"2024-09-11T08:24:18.610583+0000","last_unstale":"2024-09-11T08:24:31.307095+0000","last_undegraded":"2024-09-11T08:24:31.307095+0000","last_fullsized":"2024-09-11T08:24:31.307095+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:03.546168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069911490000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644375+0000","last_change":"2024-09-11T08:24:18.609933+0000","last_active":"2024-09-11T08:24:31.644375+0000","last_peered":"2024-09-11T08:24:31.644375+0000","last_clean":"2024-09-11T08:24:31.644375+0000","last_became_active":"2024-09-11T08:24:18.609647+0000","last_became_peered":"2024-09-11T08:24:18.609647+0000","last_unstale":"2024-09-11T08:24:31.644375+0000","last_undegraded":"2024-09-11T08:24:31.644375+0000","last_fullsized":"2024-09-11T08:24:31.644375+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:02:39.161767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"62'79","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:24:57.026649+0000","last_change":"2024-09-11T08:24:04.836167+0000","last_active":"2024-09-11T08:24:57.026649+0000","last_peered":"2024-09-11T08:24:57.026649+0000","last_clean":"2024-09-11T08:24:57.026649+0000","last_became_active":"2024-09-11T08:24:04.830418+0000","last_became_peered":"2024-09-11T08:24:04.830418+0000","last_unstale":"2024-09-11T08:24:57.026649+0000","last_undegraded":"2024-09-11T08:24:57.026649+0000","last_fullsized":"2024-09-11T08:24:57.026649+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:09.006257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:09.006257+0000","last_clean_scrub_stamp":"2024-09-11T08:22:09.006257+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:37.204656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645044+0000","last_change":"2024-09-11T08:24:22.619513+0000","last_active":"2024-09-11T08:24:31.645044+0000","last_peered":"2024-09-11T08:24:31.645044+0000","last_clean":"2024-09-11T08:24:31.645044+0000","last_became_active":"2024-09-11T08:24:22.619040+0000","last_became_peered":"2024-09-11T08:24:22.619040+0000","last_unstale":"2024-09-11T08:24:31.645044+0000","last_undegraded":"2024-09-11T08:24:31.645044+0000","last_fullsized":"2024-09-11T08:24:31.645044+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:10.597038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645238+0000","last_change":"2024-09-11T08:24:24.632729+0000","last_active":"2024-09-11T08:24:31.645238+0000","last_peered":"2024-09-11T08:24:31.645238+0000","last_clean":"2024-09-11T08:24:31.645238+0000","last_became_active":"2024-09-11T08:24:24.632418+0000","last_became_peered":"2024-09-11T08:24:24.632418+0000","last_unstale":"2024-09-11T08:24:31.645238+0000","last_undegraded":"2024-09-11T08:24:31.645238+0000","last_fullsized":"2024-09-11T08:24:31.645238+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:15:39.234973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"60'19","reported_seq":47,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645136+0000","last_change":"2024-09-11T08:24:20.978478+0000","last_active":"2024-09-11T08:24:31.645136+0000","last_peered":"2024-09-11T08:24:31.645136+0000","last_clean":"2024-09-11T08:24:31.645136+0000","last_became_active":"2024-09-11T08:24:20.977881+0000","last_became_peered":"2024-09-11T08:24:20.977881+0000","last_unstale":"2024-09-11T08:24:31.645136+0000","last_undegraded":"2024-09-11T08:24:31.645136+0000","last_fullsized":"2024-09-11T08:24:31.645136+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:27.400531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122919+0000","last_change":"2024-09-11T08:24:18.603045+0000","last_active":"2024-09-11T08:24:31.122919+0000","last_peered":"2024-09-11T08:24:31.122919+0000","last_clean":"2024-09-11T08:24:31.122919+0000","last_became_active":"2024-09-11T08:24:18.602771+0000","last_became_peered":"2024-09-11T08:24:18.602771+0000","last_unstale":"2024-09-11T08:24:31.122919+0000","last_undegraded":"2024-09-11T08:24:31.122919+0000","last_fullsized":"2024-09-11T08:24:31.122919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:44:09.301518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120653+0000","last_change":"2024-09-11T08:24:22.623300+0000","last_active":"2024-09-11T08:24:31.120653+0000","last_peered":"2024-09-11T08:24:31.120653+0000","last_clean":"2024-09-11T08:24:31.120653+0000","last_became_active":"2024-09-11T08:24:22.623034+0000","last_became_peered":"2024-09-11T08:24:22.623034+0000","last_unstale":"2024-09-11T08:24:31.120653+0000","last_undegraded":"2024-09-11T08:24:31.120653+0000","last_fullsized":"2024-09-11T08:24:31.120653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:31.553298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643865+0000","last_change":"2024-09-11T08:24:24.635990+0000","last_active":"2024-09-11T08:24:31.643865+0000","last_peered":"2024-09-11T08:24:31.643865+0000","last_clean":"2024-09-11T08:24:31.643865+0000","last_became_active":"2024-09-11T08:24:24.635467+0000","last_became_peered":"2024-09-11T08:24:24.635467+0000","last_unstale":"2024-09-11T08:24:31.643865+0000","last_undegraded":"2024-09-11T08:24:31.643865+0000","last_fullsized":"2024-09-11T08:24:31.643865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:22:46.812249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121773+0000","last_change":"2024-09-11T08:24:20.612108+0000","last_active":"2024-09-11T08:24:31.121773+0000","last_peered":"2024-09-11T08:24:31.121773+0000","last_clean":"2024-09-11T08:24:31.121773+0000","last_became_active":"2024-09-11T08:24:20.610804+0000","last_became_peered":"2024-09-11T08:24:20.610804+0000","last_unstale":"2024-09-11T08:24:31.121773+0000","last_undegraded":"2024-09-11T08:24:31.121773+0000","last_fullsized":"2024-09-11T08:24:31.121773+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:43:42.505907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644599+0000","last_change":"2024-09-11T08:24:18.607092+0000","last_active":"2024-09-11T08:24:31.644599+0000","last_peered":"2024-09-11T08:24:31.644599+0000","last_clean":"2024-09-11T08:24:31.644599+0000","last_became_active":"2024-09-11T08:24:18.606797+0000","last_became_peered":"2024-09-11T08:24:18.606797+0000","last_unstale":"2024-09-11T08:24:31.644599+0000","last_undegraded":"2024-09-11T08:24:31.644599+0000","last_fullsized":"2024-09-11T08:24:31.644599+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:46.588144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121019+0000","last_change":"2024-09-11T08:24:22.627217+0000","last_active":"2024-09-11T08:24:31.121019+0000","last_peered":"2024-09-11T08:24:31.121019+0000","last_clean":"2024-09-11T08:24:31.121019+0000","last_became_active":"2024-09-11T08:24:22.626916+0000","last_became_peered":"2024-09-11T08:24:22.626916+0000","last_unstale":"2024-09-11T08:24:31.121019+0000","last_undegraded":"2024-09-11T08:24:31.121019+0000","last_fullsized":"2024-09-11T08:24:31.121019+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:32:50.766107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"60'1","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121150+0000","last_change":"2024-09-11T08:24:24.633231+0000","last_active":"2024-09-11T08:24:31.121150+0000","last_peered":"2024-09-11T08:24:31.121150+0000","last_clean":"2024-09-11T08:24:31.121150+0000","last_became_active":"2024-09-11T08:24:24.632906+0000","last_became_peered":"2024-09-11T08:24:24.632906+0000","last_unstale":"2024-09-11T08:24:31.121150+0000","last_undegraded":"2024-09-11T08:24:31.121150+0000","last_fullsized":"2024-09-11T08:24:31.121150+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:17.280581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643579+0000","last_change":"2024-09-11T08:24:20.979035+0000","last_active":"2024-09-11T08:24:31.643579+0000","last_peered":"2024-09-11T08:24:31.643579+0000","last_clean":"2024-09-11T08:24:31.643579+0000","last_became_active":"2024-09-11T08:24:20.978730+0000","last_became_peered":"2024-09-11T08:24:20.978730+0000","last_unstale":"2024-09-11T08:24:31.643579+0000","last_undegraded":"2024-09-11T08:24:31.643579+0000","last_fullsized":"2024-09-11T08:24:31.643579+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:40.965159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121681+0000","last_change":"2024-09-11T08:24:18.604560+0000","last_active":"2024-09-11T08:24:31.121681+0000","last_peered":"2024-09-11T08:24:31.121681+0000","last_clean":"2024-09-11T08:24:31.121681+0000","last_became_active":"2024-09-11T08:24:18.603758+0000","last_became_peered":"2024-09-11T08:24:18.603758+0000","last_unstale":"2024-09-11T08:24:31.121681+0000","last_undegraded":"2024-09-11T08:24:31.121681+0000","last_fullsized":"2024-09-11T08:24:31.121681+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:40:21.684231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690344+0000","last_change":"2024-09-11T08:24:22.629891+0000","last_active":"2024-09-11T08:25:31.690344+0000","last_peered":"2024-09-11T08:25:31.690344+0000","last_clean":"2024-09-11T08:25:31.690344+0000","last_became_active":"2024-09-11T08:24:22.629368+0000","last_became_peered":"2024-09-11T08:24:22.629368+0000","last_unstale":"2024-09-11T08:25:31.690344+0000","last_undegraded":"2024-09-11T08:25:31.690344+0000","last_fullsized":"2024-09-11T08:25:31.690344+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:17:33.583486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644581+0000","last_change":"2024-09-11T08:24:25.336555+0000","last_active":"2024-09-11T08:24:31.644581+0000","last_peered":"2024-09-11T08:24:31.644581+0000","last_clean":"2024-09-11T08:24:31.644581+0000","last_became_active":"2024-09-11T08:24:25.336253+0000","last_became_peered":"2024-09-11T08:24:25.336253+0000","last_unstale":"2024-09-11T08:24:31.644581+0000","last_undegraded":"2024-09-11T08:24:31.644581+0000","last_fullsized":"2024-09-11T08:24:31.644581+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:47:38.003757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"60'12","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121094+0000","last_change":"2024-09-11T08:24:20.614181+0000","last_active":"2024-09-11T08:24:31.121094+0000","last_peered":"2024-09-11T08:24:31.121094+0000","last_clean":"2024-09-11T08:24:31.121094+0000","last_became_active":"2024-09-11T08:24:20.613848+0000","last_became_peered":"2024-09-11T08:24:20.613848+0000","last_unstale":"2024-09-11T08:24:31.121094+0000","last_undegraded":"2024-09-11T08:24:31.121094+0000","last_fullsized":"2024-09-11T08:24:31.121094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:50:38.254311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"53'2","reported_seq":34,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121910+0000","last_change":"2024-09-11T08:24:18.612162+0000","last_active":"2024-09-11T08:24:31.121910+0000","last_peered":"2024-09-11T08:24:31.121910+0000","last_clean":"2024-09-11T08:24:31.121910+0000","last_became_active":"2024-09-11T08:24:18.611899+0000","last_became_peered":"2024-09-11T08:24:18.611899+0000","last_unstale":"2024-09-11T08:24:31.121910+0000","last_undegraded":"2024-09-11T08:24:31.121910+0000","last_fullsized":"2024-09-11T08:24:31.121910+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:17.821188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120559+0000","last_change":"2024-09-11T08:24:22.616809+0000","last_active":"2024-09-11T08:24:31.120559+0000","last_peered":"2024-09-11T08:24:31.120559+0000","last_clean":"2024-09-11T08:24:31.120559+0000","last_became_active":"2024-09-11T08:24:22.616482+0000","last_became_peered":"2024-09-11T08:24:22.616482+0000","last_unstale":"2024-09-11T08:24:31.120559+0000","last_undegraded":"2024-09-11T08:24:31.120559+0000","last_fullsized":"2024-09-11T08:24:31.120559+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:14:34.651713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306890+0000","last_change":"2024-09-11T08:24:24.629705+0000","last_active":"2024-09-11T08:24:31.306890+0000","last_peered":"2024-09-11T08:24:31.306890+0000","last_clean":"2024-09-11T08:24:31.306890+0000","last_became_active":"2024-09-11T08:24:24.629423+0000","last_became_peered":"2024-09-11T08:24:24.629423+0000","last_unstale":"2024-09-11T08:24:31.306890+0000","last_undegraded":"2024-09-11T08:24:31.306890+0000","last_fullsized":"2024-09-11T08:24:31.306890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:33.829844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122087+0000","last_change":"2024-09-11T08:24:20.978644+0000","last_active":"2024-09-11T08:24:31.122087+0000","last_peered":"2024-09-11T08:24:31.122087+0000","last_clean":"2024-09-11T08:24:31.122087+0000","last_became_active":"2024-09-11T08:24:20.978167+0000","last_became_peered":"2024-09-11T08:24:20.978167+0000","last_unstale":"2024-09-11T08:24:31.122087+0000","last_undegraded":"2024-09-11T08:24:31.122087+0000","last_fullsized":"2024-09-11T08:24:31.122087+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:23.312183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120708+0000","last_change":"2024-09-11T08:24:18.606219+0000","last_active":"2024-09-11T08:24:31.120708+0000","last_peered":"2024-09-11T08:24:31.120708+0000","last_clean":"2024-09-11T08:24:31.120708+0000","last_became_active":"2024-09-11T08:24:18.605954+0000","last_became_peered":"2024-09-11T08:24:18.605954+0000","last_unstale":"2024-09-11T08:24:31.120708+0000","last_undegraded":"2024-09-11T08:24:31.120708+0000","last_fullsized":"2024-09-11T08:24:31.120708+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:04.850766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307449+0000","last_change":"2024-09-11T08:24:22.626701+0000","last_active":"2024-09-11T08:24:31.307449+0000","last_peered":"2024-09-11T08:24:31.307449+0000","last_clean":"2024-09-11T08:24:31.307449+0000","last_became_active":"2024-09-11T08:24:22.626351+0000","last_became_peered":"2024-09-11T08:24:22.626351+0000","last_unstale":"2024-09-11T08:24:31.307449+0000","last_undegraded":"2024-09-11T08:24:31.307449+0000","last_fullsized":"2024-09-11T08:24:31.307449+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:44.427374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122010+0000","last_change":"2024-09-11T08:24:24.638035+0000","last_active":"2024-09-11T08:24:31.122010+0000","last_peered":"2024-09-11T08:24:31.122010+0000","last_clean":"2024-09-11T08:24:31.122010+0000","last_became_active":"2024-09-11T08:24:24.637835+0000","last_became_peered":"2024-09-11T08:24:24.637835+0000","last_unstale":"2024-09-11T08:24:31.122010+0000","last_undegraded":"2024-09-11T08:24:31.122010+0000","last_fullsized":"2024-09-11T08:24:31.122010+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:47:54.013004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644045+0000","last_change":"2024-09-11T08:24:18.607733+0000","last_active":"2024-09-11T08:24:31.644045+0000","last_peered":"2024-09-11T08:24:31.644045+0000","last_clean":"2024-09-11T08:24:31.644045+0000","last_became_active":"2024-09-11T08:24:18.607435+0000","last_became_peered":"2024-09-11T08:24:18.607435+0000","last_unstale":"2024-09-11T08:24:31.644045+0000","last_undegraded":"2024-09-11T08:24:31.644045+0000","last_fullsized":"2024-09-11T08:24:31.644045+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:37:46.670160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"60'6","reported_seq":25,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307560+0000","last_change":"2024-09-11T08:24:20.979607+0000","last_active":"2024-09-11T08:24:31.307560+0000","last_peered":"2024-09-11T08:24:31.307560+0000","last_clean":"2024-09-11T08:24:31.307560+0000","last_became_active":"2024-09-11T08:24:20.979361+0000","last_became_peered":"2024-09-11T08:24:20.979361+0000","last_unstale":"2024-09-11T08:24:31.307560+0000","last_undegraded":"2024-09-11T08:24:31.307560+0000","last_fullsized":"2024-09-11T08:24:31.307560+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:01.257490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644976+0000","last_change":"2024-09-11T08:24:22.625848+0000","last_active":"2024-09-11T08:24:31.644976+0000","last_peered":"2024-09-11T08:24:31.644976+0000","last_clean":"2024-09-11T08:24:31.644976+0000","last_became_active":"2024-09-11T08:24:22.625511+0000","last_became_peered":"2024-09-11T08:24:22.625511+0000","last_unstale":"2024-09-11T08:24:31.644976+0000","last_undegraded":"2024-09-11T08:24:31.644976+0000","last_fullsized":"2024-09-11T08:24:31.644976+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:12:48.901989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645340+0000","last_change":"2024-09-11T08:24:25.338198+0000","last_active":"2024-09-11T08:24:31.645340+0000","last_peered":"2024-09-11T08:24:31.645340+0000","last_clean":"2024-09-11T08:24:31.645340+0000","last_became_active":"2024-09-11T08:24:25.337825+0000","last_became_peered":"2024-09-11T08:24:25.337825+0000","last_unstale":"2024-09-11T08:24:31.645340+0000","last_undegraded":"2024-09-11T08:24:31.645340+0000","last_fullsized":"2024-09-11T08:24:31.645340+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:35:13.673933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307479+0000","last_change":"2024-09-11T08:24:20.978174+0000","last_active":"2024-09-11T08:24:31.307479+0000","last_peered":"2024-09-11T08:24:31.307479+0000","last_clean":"2024-09-11T08:24:31.307479+0000","last_became_active":"2024-09-11T08:24:20.977918+0000","last_became_peered":"2024-09-11T08:24:20.977918+0000","last_unstale":"2024-09-11T08:24:31.307479+0000","last_undegraded":"2024-09-11T08:24:31.307479+0000","last_fullsized":"2024-09-11T08:24:31.307479+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:43.472987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644837+0000","last_change":"2024-09-11T08:24:18.606855+0000","last_active":"2024-09-11T08:24:31.644837+0000","last_peered":"2024-09-11T08:24:31.644837+0000","last_clean":"2024-09-11T08:24:31.644837+0000","last_became_active":"2024-09-11T08:24:18.606502+0000","last_became_peered":"2024-09-11T08:24:18.606502+0000","last_unstale":"2024-09-11T08:24:31.644837+0000","last_undegraded":"2024-09-11T08:24:31.644837+0000","last_fullsized":"2024-09-11T08:24:31.644837+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:48:30.487464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690339+0000","last_change":"2024-09-11T08:24:22.623958+0000","last_active":"2024-09-11T08:25:31.690339+0000","last_peered":"2024-09-11T08:25:31.690339+0000","last_clean":"2024-09-11T08:25:31.690339+0000","last_became_active":"2024-09-11T08:24:22.623620+0000","last_became_peered":"2024-09-11T08:24:22.623620+0000","last_unstale":"2024-09-11T08:25:31.690339+0000","last_undegraded":"2024-09-11T08:25:31.690339+0000","last_fullsized":"2024-09-11T08:25:31.690339+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:51:10.471040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121597+0000","last_change":"2024-09-11T08:24:24.633210+0000","last_active":"2024-09-11T08:24:31.121597+0000","last_peered":"2024-09-11T08:24:31.121597+0000","last_clean":"2024-09-11T08:24:31.121597+0000","last_became_active":"2024-09-11T08:24:24.632820+0000","last_became_peered":"2024-09-11T08:24:24.632820+0000","last_unstale":"2024-09-11T08:24:31.121597+0000","last_undegraded":"2024-09-11T08:24:31.121597+0000","last_fullsized":"2024-09-11T08:24:31.121597+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:48.776181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645101+0000","last_change":"2024-09-11T08:24:20.978176+0000","last_active":"2024-09-11T08:24:31.645101+0000","last_peered":"2024-09-11T08:24:31.645101+0000","last_clean":"2024-09-11T08:24:31.645101+0000","last_became_active":"2024-09-11T08:24:20.977887+0000","last_became_peered":"2024-09-11T08:24:20.977887+0000","last_unstale":"2024-09-11T08:24:31.645101+0000","last_undegraded":"2024-09-11T08:24:31.645101+0000","last_fullsized":"2024-09-11T08:24:31.645101+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:54:35.258534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121940+0000","last_change":"2024-09-11T08:24:18.606550+0000","last_active":"2024-09-11T08:24:31.121940+0000","last_peered":"2024-09-11T08:24:31.121940+0000","last_clean":"2024-09-11T08:24:31.121940+0000","last_became_active":"2024-09-11T08:24:18.606274+0000","last_became_peered":"2024-09-11T08:24:18.606274+0000","last_unstale":"2024-09-11T08:24:31.121940+0000","last_undegraded":"2024-09-11T08:24:31.121940+0000","last_fullsized":"2024-09-11T08:24:31.121940+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:15:44.540152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"60'8","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690426+0000","last_change":"2024-09-11T08:24:22.626647+0000","last_active":"2024-09-11T08:25:31.690426+0000","last_peered":"2024-09-11T08:25:31.690426+0000","last_clean":"2024-09-11T08:25:31.690426+0000","last_became_active":"2024-09-11T08:24:22.626296+0000","last_became_peered":"2024-09-11T08:24:22.626296+0000","last_unstale":"2024-09-11T08:25:31.690426+0000","last_undegraded":"2024-09-11T08:25:31.690426+0000","last_fullsized":"2024-09-11T08:25:31.690426+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:45:26.487207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121806+0000","last_change":"2024-09-11T08:24:24.634883+0000","last_active":"2024-09-11T08:24:31.121806+0000","last_peered":"2024-09-11T08:24:31.121806+0000","last_clean":"2024-09-11T08:24:31.121806+0000","last_became_active":"2024-09-11T08:24:24.634571+0000","last_became_peered":"2024-09-11T08:24:24.634571+0000","last_unstale":"2024-09-11T08:24:31.121806+0000","last_undegraded":"2024-09-11T08:24:31.121806+0000","last_fullsized":"2024-09-11T08:24:31.121806+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:56:03.091789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643824+0000","last_change":"2024-09-11T08:24:20.608916+0000","last_active":"2024-09-11T08:24:31.643824+0000","last_peered":"2024-09-11T08:24:31.643824+0000","last_clean":"2024-09-11T08:24:31.643824+0000","last_became_active":"2024-09-11T08:24:20.608642+0000","last_became_peered":"2024-09-11T08:24:20.608642+0000","last_unstale":"2024-09-11T08:24:31.643824+0000","last_undegraded":"2024-09-11T08:24:31.643824+0000","last_fullsized":"2024-09-11T08:24:31.643824+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:58.368361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644068+0000","last_change":"2024-09-11T08:24:18.610475+0000","last_active":"2024-09-11T08:24:31.644068+0000","last_peered":"2024-09-11T08:24:31.644068+0000","last_clean":"2024-09-11T08:24:31.644068+0000","last_became_active":"2024-09-11T08:24:18.610205+0000","last_became_peered":"2024-09-11T08:24:18.610205+0000","last_unstale":"2024-09-11T08:24:31.644068+0000","last_undegraded":"2024-09-11T08:24:31.644068+0000","last_fullsized":"2024-09-11T08:24:31.644068+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:05.787800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307014+0000","last_change":"2024-09-11T08:24:22.630585+0000","last_active":"2024-09-11T08:24:31.307014+0000","last_peered":"2024-09-11T08:24:31.307014+0000","last_clean":"2024-09-11T08:24:31.307014+0000","last_became_active":"2024-09-11T08:24:22.630316+0000","last_became_peered":"2024-09-11T08:24:22.630316+0000","last_unstale":"2024-09-11T08:24:31.307014+0000","last_undegraded":"2024-09-11T08:24:31.307014+0000","last_fullsized":"2024-09-11T08:24:31.307014+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:08:13.012767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120812+0000","last_change":"2024-09-11T08:24:24.632120+0000","last_active":"2024-09-11T08:24:31.120812+0000","last_peered":"2024-09-11T08:24:31.120812+0000","last_clean":"2024-09-11T08:24:31.120812+0000","last_became_active":"2024-09-11T08:24:24.631620+0000","last_became_peered":"2024-09-11T08:24:24.631620+0000","last_unstale":"2024-09-11T08:24:31.120812+0000","last_undegraded":"2024-09-11T08:24:31.120812+0000","last_fullsized":"2024-09-11T08:24:31.120812+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:39.994685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121202+0000","last_change":"2024-09-11T08:24:20.606969+0000","last_active":"2024-09-11T08:24:31.121202+0000","last_peered":"2024-09-11T08:24:31.121202+0000","last_clean":"2024-09-11T08:24:31.121202+0000","last_became_active":"2024-09-11T08:24:20.606406+0000","last_became_peered":"2024-09-11T08:24:20.606406+0000","last_unstale":"2024-09-11T08:24:31.121202+0000","last_undegraded":"2024-09-11T08:24:31.121202+0000","last_fullsized":"2024-09-11T08:24:31.121202+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:01:26.826903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644643+0000","last_change":"2024-09-11T08:24:18.607479+0000","last_active":"2024-09-11T08:24:31.644643+0000","last_peered":"2024-09-11T08:24:31.644643+0000","last_clean":"2024-09-11T08:24:31.644643+0000","last_became_active":"2024-09-11T08:24:18.606062+0000","last_became_peered":"2024-09-11T08:24:18.606062+0000","last_unstale":"2024-09-11T08:24:31.644643+0000","last_undegraded":"2024-09-11T08:24:31.644643+0000","last_fullsized":"2024-09-11T08:24:31.644643+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:07:38.453373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643734+0000","last_change":"2024-09-11T08:24:22.625180+0000","last_active":"2024-09-11T08:24:31.643734+0000","last_peered":"2024-09-11T08:24:31.643734+0000","last_clean":"2024-09-11T08:24:31.643734+0000","last_became_active":"2024-09-11T08:24:22.624879+0000","last_became_peered":"2024-09-11T08:24:22.624879+0000","last_unstale":"2024-09-11T08:24:31.643734+0000","last_undegraded":"2024-09-11T08:24:31.643734+0000","last_fullsized":"2024-09-11T08:24:31.643734+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:13.770008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306586+0000","last_change":"2024-09-11T08:24:24.635962+0000","last_active":"2024-09-11T08:24:31.306586+0000","last_peered":"2024-09-11T08:24:31.306586+0000","last_clean":"2024-09-11T08:24:31.306586+0000","last_became_active":"2024-09-11T08:24:24.635651+0000","last_became_peered":"2024-09-11T08:24:24.635651+0000","last_unstale":"2024-09-11T08:24:31.306586+0000","last_undegraded":"2024-09-11T08:24:31.306586+0000","last_fullsized":"2024-09-11T08:24:31.306586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:12:07.881390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"60'4","reported_seq":22,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306586+0000","last_change":"2024-09-11T08:24:20.614336+0000","last_active":"2024-09-11T08:24:31.306586+0000","last_peered":"2024-09-11T08:24:31.306586+0000","last_clean":"2024-09-11T08:24:31.306586+0000","last_became_active":"2024-09-11T08:24:20.614082+0000","last_became_peered":"2024-09-11T08:24:20.614082+0000","last_unstale":"2024-09-11T08:24:31.306586+0000","last_undegraded":"2024-09-11T08:24:31.306586+0000","last_fullsized":"2024-09-11T08:24:31.306586+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:52:48.491779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645103+0000","last_change":"2024-09-11T08:24:18.610784+0000","last_active":"2024-09-11T08:24:31.645103+0000","last_peered":"2024-09-11T08:24:31.645103+0000","last_clean":"2024-09-11T08:24:31.645103+0000","last_became_active":"2024-09-11T08:24:18.610524+0000","last_became_peered":"2024-09-11T08:24:18.610524+0000","last_unstale":"2024-09-11T08:24:31.645103+0000","last_undegraded":"2024-09-11T08:24:31.645103+0000","last_fullsized":"2024-09-11T08:24:31.645103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:18.290089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643853+0000","last_change":"2024-09-11T08:24:22.625145+0000","last_active":"2024-09-11T08:24:31.643853+0000","last_peered":"2024-09-11T08:24:31.643853+0000","last_clean":"2024-09-11T08:24:31.643853+0000","last_became_active":"2024-09-11T08:24:22.624888+0000","last_became_peered":"2024-09-11T08:24:22.624888+0000","last_unstale":"2024-09-11T08:24:31.643853+0000","last_undegraded":"2024-09-11T08:24:31.643853+0000","last_fullsized":"2024-09-11T08:24:31.643853+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:11:19.576615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644112+0000","last_change":"2024-09-11T08:24:24.635266+0000","last_active":"2024-09-11T08:24:31.644112+0000","last_peered":"2024-09-11T08:24:31.644112+0000","last_clean":"2024-09-11T08:24:31.644112+0000","last_became_active":"2024-09-11T08:24:24.634935+0000","last_became_peered":"2024-09-11T08:24:24.634935+0000","last_unstale":"2024-09-11T08:24:31.644112+0000","last_undegraded":"2024-09-11T08:24:31.644112+0000","last_fullsized":"2024-09-11T08:24:31.644112+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:58:05.175649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306762+0000","last_change":"2024-09-11T08:24:20.978167+0000","last_active":"2024-09-11T08:24:31.306762+0000","last_peered":"2024-09-11T08:24:31.306762+0000","last_clean":"2024-09-11T08:24:31.306762+0000","last_became_active":"2024-09-11T08:24:20.977877+0000","last_became_peered":"2024-09-11T08:24:20.977877+0000","last_unstale":"2024-09-11T08:24:31.306762+0000","last_undegraded":"2024-09-11T08:24:31.306762+0000","last_fullsized":"2024-09-11T08:24:31.306762+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:39:42.639634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644469+0000","last_change":"2024-09-11T08:24:18.614076+0000","last_active":"2024-09-11T08:24:31.644469+0000","last_peered":"2024-09-11T08:24:31.644469+0000","last_clean":"2024-09-11T08:24:31.644469+0000","last_became_active":"2024-09-11T08:24:18.613789+0000","last_became_peered":"2024-09-11T08:24:18.613789+0000","last_unstale":"2024-09-11T08:24:31.644469+0000","last_undegraded":"2024-09-11T08:24:31.644469+0000","last_fullsized":"2024-09-11T08:24:31.644469+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:39:35.120681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643479+0000","last_change":"2024-09-11T08:24:22.628532+0000","last_active":"2024-09-11T08:24:31.643479+0000","last_peered":"2024-09-11T08:24:31.643479+0000","last_clean":"2024-09-11T08:24:31.643479+0000","last_became_active":"2024-09-11T08:24:22.628211+0000","last_became_peered":"2024-09-11T08:24:22.628211+0000","last_unstale":"2024-09-11T08:24:31.643479+0000","last_undegraded":"2024-09-11T08:24:31.643479+0000","last_fullsized":"2024-09-11T08:24:31.643479+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:14.610567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306746+0000","last_change":"2024-09-11T08:24:25.338172+0000","last_active":"2024-09-11T08:24:31.306746+0000","last_peered":"2024-09-11T08:24:31.306746+0000","last_clean":"2024-09-11T08:24:31.306746+0000","last_became_active":"2024-09-11T08:24:25.337797+0000","last_became_peered":"2024-09-11T08:24:25.337797+0000","last_unstale":"2024-09-11T08:24:31.306746+0000","last_undegraded":"2024-09-11T08:24:31.306746+0000","last_fullsized":"2024-09-11T08:24:31.306746+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:35:20.444353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"60'1","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644966+0000","last_change":"2024-09-11T08:24:24.635456+0000","last_active":"2024-09-11T08:24:31.644966+0000","last_peered":"2024-09-11T08:24:31.644966+0000","last_clean":"2024-09-11T08:24:31.644966+0000","last_became_active":"2024-09-11T08:24:24.635167+0000","last_became_peered":"2024-09-11T08:24:24.635167+0000","last_unstale":"2024-09-11T08:24:31.644966+0000","last_undegraded":"2024-09-11T08:24:31.644966+0000","last_fullsized":"2024-09-11T08:24:31.644966+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:19:15.885301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643653+0000","last_change":"2024-09-11T08:24:18.605425+0000","last_active":"2024-09-11T08:24:31.643653+0000","last_peered":"2024-09-11T08:24:31.643653+0000","last_clean":"2024-09-11T08:24:31.643653+0000","last_became_active":"2024-09-11T08:24:18.604978+0000","last_became_peered":"2024-09-11T08:24:18.604978+0000","last_unstale":"2024-09-11T08:24:31.643653+0000","last_undegraded":"2024-09-11T08:24:31.643653+0000","last_fullsized":"2024-09-11T08:24:31.643653+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:03:33.609832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122019+0000","last_change":"2024-09-11T08:24:20.980359+0000","last_active":"2024-09-11T08:24:31.122019+0000","last_peered":"2024-09-11T08:24:31.122019+0000","last_clean":"2024-09-11T08:24:31.122019+0000","last_became_active":"2024-09-11T08:24:20.980095+0000","last_became_peered":"2024-09-11T08:24:20.980095+0000","last_unstale":"2024-09-11T08:24:31.122019+0000","last_undegraded":"2024-09-11T08:24:31.122019+0000","last_fullsized":"2024-09-11T08:24:31.122019+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:52:00.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"60'8","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690377+0000","last_change":"2024-09-11T08:24:22.625357+0000","last_active":"2024-09-11T08:25:31.690377+0000","last_peered":"2024-09-11T08:25:31.690377+0000","last_clean":"2024-09-11T08:25:31.690377+0000","last_became_active":"2024-09-11T08:24:22.623565+0000","last_became_peered":"2024-09-11T08:24:22.623565+0000","last_unstale":"2024-09-11T08:25:31.690377+0000","last_undegraded":"2024-09-11T08:25:31.690377+0000","last_fullsized":"2024-09-11T08:25:31.690377+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:45:24.474162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644038+0000","last_change":"2024-09-11T08:24:24.630905+0000","last_active":"2024-09-11T08:24:31.644038+0000","last_peered":"2024-09-11T08:24:31.644038+0000","last_clean":"2024-09-11T08:24:31.644038+0000","last_became_active":"2024-09-11T08:24:24.630613+0000","last_became_peered":"2024-09-11T08:24:24.630613+0000","last_unstale":"2024-09-11T08:24:31.644038+0000","last_undegraded":"2024-09-11T08:24:31.644038+0000","last_fullsized":"2024-09-11T08:24:31.644038+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:15:02.340916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307036+0000","last_change":"2024-09-11T08:24:18.609433+0000","last_active":"2024-09-11T08:24:31.307036+0000","last_peered":"2024-09-11T08:24:31.307036+0000","last_clean":"2024-09-11T08:24:31.307036+0000","last_became_active":"2024-09-11T08:24:18.608650+0000","last_became_peered":"2024-09-11T08:24:18.608650+0000","last_unstale":"2024-09-11T08:24:31.307036+0000","last_undegraded":"2024-09-11T08:24:31.307036+0000","last_fullsized":"2024-09-11T08:24:31.307036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:31:13.399425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644392+0000","last_change":"2024-09-11T08:24:20.611153+0000","last_active":"2024-09-11T08:24:31.644392+0000","last_peered":"2024-09-11T08:24:31.644392+0000","last_clean":"2024-09-11T08:24:31.644392+0000","last_became_active":"2024-09-11T08:24:20.610802+0000","last_became_peered":"2024-09-11T08:24:20.610802+0000","last_unstale":"2024-09-11T08:24:31.644392+0000","last_undegraded":"2024-09-11T08:24:31.644392+0000","last_fullsized":"2024-09-11T08:24:31.644392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:47:48.881371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122024+0000","last_change":"2024-09-11T08:24:22.623152+0000","last_active":"2024-09-11T08:24:31.122024+0000","last_peered":"2024-09-11T08:24:31.122024+0000","last_clean":"2024-09-11T08:24:31.122024+0000","last_became_active":"2024-09-11T08:24:22.622845+0000","last_became_peered":"2024-09-11T08:24:22.622845+0000","last_unstale":"2024-09-11T08:24:31.122024+0000","last_undegraded":"2024-09-11T08:24:31.122024+0000","last_fullsized":"2024-09-11T08:24:31.122024+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:57:28.315295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121603+0000","last_change":"2024-09-11T08:24:25.338725+0000","last_active":"2024-09-11T08:24:31.121603+0000","last_peered":"2024-09-11T08:24:31.121603+0000","last_clean":"2024-09-11T08:24:31.121603+0000","last_became_active":"2024-09-11T08:24:25.338450+0000","last_became_peered":"2024-09-11T08:24:25.338450+0000","last_unstale":"2024-09-11T08:24:31.121603+0000","last_undegraded":"2024-09-11T08:24:31.121603+0000","last_fullsized":"2024-09-11T08:24:31.121603+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:22:27.532897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645215+0000","last_change":"2024-09-11T08:24:18.609876+0000","last_active":"2024-09-11T08:24:31.645215+0000","last_peered":"2024-09-11T08:24:31.645215+0000","last_clean":"2024-09-11T08:24:31.645215+0000","last_became_active":"2024-09-11T08:24:18.608554+0000","last_became_peered":"2024-09-11T08:24:18.608554+0000","last_unstale":"2024-09-11T08:24:31.645215+0000","last_undegraded":"2024-09-11T08:24:31.645215+0000","last_fullsized":"2024-09-11T08:24:31.645215+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:06:20.895951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122255+0000","last_change":"2024-09-11T08:24:20.613552+0000","last_active":"2024-09-11T08:24:31.122255+0000","last_peered":"2024-09-11T08:24:31.122255+0000","last_clean":"2024-09-11T08:24:31.122255+0000","last_became_active":"2024-09-11T08:24:20.613263+0000","last_became_peered":"2024-09-11T08:24:20.613263+0000","last_unstale":"2024-09-11T08:24:31.122255+0000","last_undegraded":"2024-09-11T08:24:31.122255+0000","last_fullsized":"2024-09-11T08:24:31.122255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:06:27.820868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644232+0000","last_change":"2024-09-11T08:24:22.629394+0000","last_active":"2024-09-11T08:24:31.644232+0000","last_peered":"2024-09-11T08:24:31.644232+0000","last_clean":"2024-09-11T08:24:31.644232+0000","last_became_active":"2024-09-11T08:24:22.629114+0000","last_became_peered":"2024-09-11T08:24:22.629114+0000","last_unstale":"2024-09-11T08:24:31.644232+0000","last_undegraded":"2024-09-11T08:24:31.644232+0000","last_fullsized":"2024-09-11T08:24:31.644232+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:57:42.765458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307488+0000","last_change":"2024-09-11T08:24:25.336832+0000","last_active":"2024-09-11T08:24:31.307488+0000","last_peered":"2024-09-11T08:24:31.307488+0000","last_clean":"2024-09-11T08:24:31.307488+0000","last_became_active":"2024-09-11T08:24:25.336466+0000","last_became_peered":"2024-09-11T08:24:25.336466+0000","last_unstale":"2024-09-11T08:24:31.307488+0000","last_undegraded":"2024-09-11T08:24:31.307488+0000","last_fullsized":"2024-09-11T08:24:31.307488+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:35.627909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307148+0000","last_change":"2024-09-11T08:24:18.614611+0000","last_active":"2024-09-11T08:24:31.307148+0000","last_peered":"2024-09-11T08:24:31.307148+0000","last_clean":"2024-09-11T08:24:31.307148+0000","last_became_active":"2024-09-11T08:24:18.614295+0000","last_became_peered":"2024-09-11T08:24:18.614295+0000","last_unstale":"2024-09-11T08:24:31.307148+0000","last_undegraded":"2024-09-11T08:24:31.307148+0000","last_fullsized":"2024-09-11T08:24:31.307148+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:10:21.065116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"60'12","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307206+0000","last_change":"2024-09-11T08:24:20.616441+0000","last_active":"2024-09-11T08:24:31.307206+0000","last_peered":"2024-09-11T08:24:31.307206+0000","last_clean":"2024-09-11T08:24:31.307206+0000","last_became_active":"2024-09-11T08:24:20.616151+0000","last_became_peered":"2024-09-11T08:24:20.616151+0000","last_unstale":"2024-09-11T08:24:31.307206+0000","last_undegraded":"2024-09-11T08:24:31.307206+0000","last_fullsized":"2024-09-11T08:24:31.307206+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:39.015560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121411+0000","last_change":"2024-09-11T08:24:22.627742+0000","last_active":"2024-09-11T08:24:31.121411+0000","last_peered":"2024-09-11T08:24:31.121411+0000","last_clean":"2024-09-11T08:24:31.121411+0000","last_became_active":"2024-09-11T08:24:22.627089+0000","last_became_peered":"2024-09-11T08:24:22.627089+0000","last_unstale":"2024-09-11T08:24:31.121411+0000","last_undegraded":"2024-09-11T08:24:31.121411+0000","last_fullsized":"2024-09-11T08:24:31.121411+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:28:46.573033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397526,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298548,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446604,"statfs":{"total":95995035648,"available":95689322496,"internally_reserved":0,"allocated":1220608,"data_stored":825318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.44400000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.47199999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.39800000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.379}]},{"osd":4,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.315},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.33200000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.40999999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38600000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593756,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298524,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446628,"statfs":{"total":95995035648,"available":95689347072,"internally_reserved":0,"allocated":1196032,"data_stored":824985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.377},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.41199999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.371},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.30399999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.255},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.28199999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.318},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32900000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.36199999999999999}]}]},{"osd":5,"up_from":37,"seq":158913789982,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":602112,"data_stored":234532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73299999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.63900000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":6.7130000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68300000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70199999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":6.7290000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.56399999999999995,"15min":0.57299999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71099999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.34599999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":6.7690000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67000000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.48199999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.375,"15min":0.36699999999999999},"min":{"1min":0.28799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66600000000000004,"5min":0.874,"15min":0.874},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38500000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.502},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.47199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.44},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.44900000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297964,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447188,"statfs":{"total":95995035648,"available":95689920512,"internally_reserved":0,"allocated":622592,"data_stored":234426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.79400000000000004,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.441},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69199999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36699999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.45200000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.376},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70799999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.42399999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.91000000000000003,"15min":0.96999999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69099999999999995,"5min":11.728999999999999,"15min":11.728999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.27600000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.311},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.28199999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.35399999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297984,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447168,"statfs":{"total":95995035648,"available":95689900032,"internally_reserved":0,"allocated":643072,"data_stored":235139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.36599999999999999,"15min":0.375},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.42399999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.372,"15min":0.38},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.499,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.40000000000000002,"15min":0.41299999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.48999999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33100000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.48799999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.315},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40400000000000003,"15min":0.40999999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.32300000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.67800000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68899999999999995,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.77600000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56599999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.66600000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58299999999999996}]},{"osd":6,"last update":"Wed Sep 11 08:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.71699999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56100000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297948,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447204,"statfs":{"total":95995035648,"available":95689936896,"internally_reserved":0,"allocated":606208,"data_stored":234429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.33600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.376},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.314}]},{"osd":1,"last update":"Wed Sep 11 08:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.39500000000000002},"min":{"1min":0.24099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.35499999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:25:35 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.435,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.51200000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40000000000000002,"15min":0.40400000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.40300000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68600000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.79900000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.625},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.626,"15min":0.58799999999999997},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.9329999999999998,"5min":5.9329999999999998,"15min":5.9329999999999998},"last":0.57299999999999995}]},{"osd":6,"last update":"Wed Sep 11 08:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67000000000000004}]},{"osd":7,"last update":"Wed Sep 11 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.976,"5min":5.976,"15min":5.976},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.56899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":236214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32000000000000001,"15min":0.308},"min":{"1min":0.191,"5min":0.19,"15min":0.19},"max":{"1min":0.47799999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.219,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.371,"15min":0.372},"min":{"1min":0.23999999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.504,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.215,"15min":0.215},"max":{"1min":0.56100000000000005,"5min":0.621,"15min":0.621},"last":0.41499999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.442,"15min":0.46000000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.624,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41699999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36899999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.70499999999999996}]},{"osd":5,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56799999999999995,"15min":0.56399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.751}]},{"osd":7,"last update":"Wed Sep 11 08:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298540,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446612,"statfs":{"total":95995035648,"available":95689330688,"internally_reserved":0,"allocated":1216512,"data_stored":824653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.309},"min":{"1min":0.20599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.498,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.34399999999999997,"15min":0.32400000000000001},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.52000000000000002,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.314}]},{"osd":2,"last update":"Wed Sep 11 08:25:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.44500000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.35399999999999998,"15min":0.34100000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.46899999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.24299999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.41999999999999998,"15min":0.43099999999999999},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.48799999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.39400000000000002,"15min":0.41099999999999998},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.44400000000000001,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.32600000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53300000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47499999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:25:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.41499999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.44700000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.40999999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:25:43.640 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-11T08:25:43.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:43 smithi139 ceph-mon[32601]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:43.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:25:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:25:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:43.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:43 smithi110 ceph-mon[28998]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:43.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:43 smithi110 ceph-mon[39404]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:44.465 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:45.403 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 podman[72602]: 2024-09-11 08:25:45.307548966 +0000 UTC m=+3.197034254 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-11T08:25:45.825 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 podman[72602]: 2024-09-11 08:25:45.53118181 +0000 UTC m=+3.420667082 container create 34f1184be6c7ccab5ec2d614a43f71b8d95e40fc80d812eb7e535266cbf039ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:25:45.825 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 podman[72602]: 2024-09-11 08:25:45.741381989 +0000 UTC m=+3.630867261 container init 34f1184be6c7ccab5ec2d614a43f71b8d95e40fc80d812eb7e535266cbf039ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:25:45.825 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 podman[72602]: 2024-09-11 08:25:45.744374954 +0000 UTC m=+3.633860226 container start 34f1184be6c7ccab5ec2d614a43f71b8d95e40fc80d812eb7e535266cbf039ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:25:45.825 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.748Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-11T08:25:45.825 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.749Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-11T08:25:45.825 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.749Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.749Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=arp 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-11T08:25:45.826 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=edac 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-11T08:25:45.827 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=os 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-11T08:25:45.828 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=stat 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=time 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=uname 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-11T08:25:45.829 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.750Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-11T08:25:45.830 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.751Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-11T08:25:45.830 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.751Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-11T08:25:45.830 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a[72789]: ts=2024-09-11T08:25:45.751Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-11T08:25:45.830 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 bash[72602]: 34f1184be6c7ccab5ec2d614a43f71b8d95e40fc80d812eb7e535266cbf039ef 2024-09-11T08:25:46.153 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:25:45 smithi110 systemd[1]: Started Ceph node-exporter.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:25:46.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[39404]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:46.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[28998]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:46.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.154 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:46 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:25:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:25:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:46 smithi139 ceph-mon[32601]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:25:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:46 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:46 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:46 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:46.790 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:46 smithi139 systemd[1]: Starting Ceph node-exporter.b for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:25:46.990 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:25:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-11T08:25:47.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:47 smithi139 ceph-mon[32601]: Deploying daemon node-exporter.b on smithi139 2024-09-11T08:25:47.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:47 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:47.072 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:46 smithi139 bash[63301]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-11T08:25:47.276 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:47 smithi110 ceph-mon[28998]: Deploying daemon node-exporter.b on smithi139 2024-09-11T08:25:47.276 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:47 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:47.276 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:47 smithi110 ceph-mon[39404]: Deploying daemon node-exporter.b on smithi139 2024-09-11T08:25:47.276 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:47 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:47.571 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:47 smithi139 bash[63301]: Getting image source signatures 2024-09-11T08:25:47.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:47 smithi139 bash[63301]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-11T08:25:47.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:47 smithi139 bash[63301]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-11T08:25:47.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:47 smithi139 bash[63301]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-11T08:25:47.861 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-11T08:25:44.978550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":926,"num_read_kb":783,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":548,"ondisk_log_size":548,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385448,"kb_used_data":6604,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575768,"statfs":{"total":767960285184,"available":765517586432,"internally_reserved":0,"allocated":6762496,"data_stored":3649696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002375"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306862+0000","last_change":"2024-09-11T08:24:25.338389+0000","last_active":"2024-09-11T08:24:31.306862+0000","last_peered":"2024-09-11T08:24:31.306862+0000","last_clean":"2024-09-11T08:24:31.306862+0000","last_became_active":"2024-09-11T08:24:25.337868+0000","last_became_peered":"2024-09-11T08:24:25.337868+0000","last_unstale":"2024-09-11T08:24:31.306862+0000","last_undegraded":"2024-09-11T08:24:31.306862+0000","last_fullsized":"2024-09-11T08:24:31.306862+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:27:32.651542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121531+0000","last_change":"2024-09-11T08:24:18.607982+0000","last_active":"2024-09-11T08:24:31.121531+0000","last_peered":"2024-09-11T08:24:31.121531+0000","last_clean":"2024-09-11T08:24:31.121531+0000","last_became_active":"2024-09-11T08:24:18.607693+0000","last_became_peered":"2024-09-11T08:24:18.607693+0000","last_unstale":"2024-09-11T08:24:31.121531+0000","last_undegraded":"2024-09-11T08:24:31.121531+0000","last_fullsized":"2024-09-11T08:24:31.121531+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:20:37.741409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306917+0000","last_change":"2024-09-11T08:24:20.616820+0000","last_active":"2024-09-11T08:24:31.306917+0000","last_peered":"2024-09-11T08:24:31.306917+0000","last_clean":"2024-09-11T08:24:31.306917+0000","last_became_active":"2024-09-11T08:24:20.616359+0000","last_became_peered":"2024-09-11T08:24:20.616359+0000","last_unstale":"2024-09-11T08:24:31.306917+0000","last_undegraded":"2024-09-11T08:24:31.306917+0000","last_fullsized":"2024-09-11T08:24:31.306917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:43.193298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120987+0000","last_change":"2024-09-11T08:24:22.622622+0000","last_active":"2024-09-11T08:24:31.120987+0000","last_peered":"2024-09-11T08:24:31.120987+0000","last_clean":"2024-09-11T08:24:31.120987+0000","last_became_active":"2024-09-11T08:24:22.622225+0000","last_became_peered":"2024-09-11T08:24:22.622225+0000","last_unstale":"2024-09-11T08:24:31.120987+0000","last_undegraded":"2024-09-11T08:24:31.120987+0000","last_fullsized":"2024-09-11T08:24:31.120987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:56.926961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307464+0000","last_change":"2024-09-11T08:24:18.606375+0000","last_active":"2024-09-11T08:24:31.307464+0000","last_peered":"2024-09-11T08:24:31.307464+0000","last_clean":"2024-09-11T08:24:31.307464+0000","last_became_active":"2024-09-11T08:24:18.606112+0000","last_became_peered":"2024-09-11T08:24:18.606112+0000","last_unstale":"2024-09-11T08:24:31.307464+0000","last_undegraded":"2024-09-11T08:24:31.307464+0000","last_fullsized":"2024-09-11T08:24:31.307464+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:40:12.874596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120996+0000","last_change":"2024-09-11T08:24:20.608184+0000","last_active":"2024-09-11T08:24:31.120996+0000","last_peered":"2024-09-11T08:24:31.120996+0000","last_clean":"2024-09-11T08:24:31.120996+0000","last_became_active":"2024-09-11T08:24:20.607942+0000","last_became_peered":"2024-09-11T08:24:20.607942+0000","last_unstale":"2024-09-11T08:24:31.120996+0000","last_undegraded":"2024-09-11T08:24:31.120996+0000","last_fullsized":"2024-09-11T08:24:31.120996+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.976179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643685+0000","last_change":"2024-09-11T08:24:22.621538+0000","last_active":"2024-09-11T08:24:31.643685+0000","last_peered":"2024-09-11T08:24:31.643685+0000","last_clean":"2024-09-11T08:24:31.643685+0000","last_became_active":"2024-09-11T08:24:22.621254+0000","last_became_peered":"2024-09-11T08:24:22.621254+0000","last_unstale":"2024-09-11T08:24:31.643685+0000","last_undegraded":"2024-09-11T08:24:31.643685+0000","last_fullsized":"2024-09-11T08:24:31.643685+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:11:18.928367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121314+0000","last_change":"2024-09-11T08:24:24.630199+0000","last_active":"2024-09-11T08:24:31.121314+0000","last_peered":"2024-09-11T08:24:31.121314+0000","last_clean":"2024-09-11T08:24:31.121314+0000","last_became_active":"2024-09-11T08:24:24.629091+0000","last_became_peered":"2024-09-11T08:24:24.629091+0000","last_unstale":"2024-09-11T08:24:31.121314+0000","last_undegraded":"2024-09-11T08:24:31.121314+0000","last_fullsized":"2024-09-11T08:24:31.121314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:01:56.296158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645560+0000","last_change":"2024-09-11T08:24:18.612232+0000","last_active":"2024-09-11T08:24:31.645560+0000","last_peered":"2024-09-11T08:24:31.645560+0000","last_clean":"2024-09-11T08:24:31.645560+0000","last_became_active":"2024-09-11T08:24:18.607618+0000","last_became_peered":"2024-09-11T08:24:18.607618+0000","last_unstale":"2024-09-11T08:24:31.645560+0000","last_undegraded":"2024-09-11T08:24:31.645560+0000","last_fullsized":"2024-09-11T08:24:31.645560+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:49:02.476467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"60'5","reported_seq":26,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121198+0000","last_change":"2024-09-11T08:24:20.614170+0000","last_active":"2024-09-11T08:24:31.121198+0000","last_peered":"2024-09-11T08:24:31.121198+0000","last_clean":"2024-09-11T08:24:31.121198+0000","last_became_active":"2024-09-11T08:24:20.613851+0000","last_became_peered":"2024-09-11T08:24:20.613851+0000","last_unstale":"2024-09-11T08:24:31.121198+0000","last_undegraded":"2024-09-11T08:24:31.121198+0000","last_fullsized":"2024-09-11T08:24:31.121198+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:43.175051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644181+0000","last_change":"2024-09-11T08:24:22.628941+0000","last_active":"2024-09-11T08:24:31.644181+0000","last_peered":"2024-09-11T08:24:31.644181+0000","last_clean":"2024-09-11T08:24:31.644181+0000","last_became_active":"2024-09-11T08:24:22.628649+0000","last_became_peered":"2024-09-11T08:24:22.628649+0000","last_unstale":"2024-09-11T08:24:31.644181+0000","last_undegraded":"2024-09-11T08:24:31.644181+0000","last_fullsized":"2024-09-11T08:24:31.644181+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:50:19.270029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643770+0000","last_change":"2024-09-11T08:24:24.636731+0000","last_active":"2024-09-11T08:24:31.643770+0000","last_peered":"2024-09-11T08:24:31.643770+0000","last_clean":"2024-09-11T08:24:31.643770+0000","last_became_active":"2024-09-11T08:24:24.636384+0000","last_became_peered":"2024-09-11T08:24:24.636384+0000","last_unstale":"2024-09-11T08:24:31.643770+0000","last_undegraded":"2024-09-11T08:24:31.643770+0000","last_fullsized":"2024-09-11T08:24:31.643770+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:46.349874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645295+0000","last_change":"2024-09-11T08:24:18.611777+0000","last_active":"2024-09-11T08:24:31.645295+0000","last_peered":"2024-09-11T08:24:31.645295+0000","last_clean":"2024-09-11T08:24:31.645295+0000","last_became_active":"2024-09-11T08:24:18.611494+0000","last_became_peered":"2024-09-11T08:24:18.611494+0000","last_unstale":"2024-09-11T08:24:31.645295+0000","last_undegraded":"2024-09-11T08:24:31.645295+0000","last_fullsized":"2024-09-11T08:24:31.645295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:24:14.345542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121393+0000","last_change":"2024-09-11T08:24:20.613540+0000","last_active":"2024-09-11T08:24:31.121393+0000","last_peered":"2024-09-11T08:24:31.121393+0000","last_clean":"2024-09-11T08:24:31.121393+0000","last_became_active":"2024-09-11T08:24:20.613272+0000","last_became_peered":"2024-09-11T08:24:20.613272+0000","last_unstale":"2024-09-11T08:24:31.121393+0000","last_undegraded":"2024-09-11T08:24:31.121393+0000","last_fullsized":"2024-09-11T08:24:31.121393+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:03.597166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644161+0000","last_change":"2024-09-11T08:24:22.624351+0000","last_active":"2024-09-11T08:24:31.644161+0000","last_peered":"2024-09-11T08:24:31.644161+0000","last_clean":"2024-09-11T08:24:31.644161+0000","last_became_active":"2024-09-11T08:24:22.623880+0000","last_became_peered":"2024-09-11T08:24:22.623880+0000","last_unstale":"2024-09-11T08:24:31.644161+0000","last_undegraded":"2024-09-11T08:24:31.644161+0000","last_fullsized":"2024-09-11T08:24:31.644161+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:26:39.388477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121099+0000","last_change":"2024-09-11T08:24:24.631898+0000","last_active":"2024-09-11T08:24:31.121099+0000","last_peered":"2024-09-11T08:24:31.121099+0000","last_clean":"2024-09-11T08:24:31.121099+0000","last_became_active":"2024-09-11T08:24:24.631310+0000","last_became_peered":"2024-09-11T08:24:24.631310+0000","last_unstale":"2024-09-11T08:24:31.121099+0000","last_undegraded":"2024-09-11T08:24:31.121099+0000","last_fullsized":"2024-09-11T08:24:31.121099+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:24:27.906608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"60'17","reported_seq":44,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121646+0000","last_change":"2024-09-11T08:24:20.606737+0000","last_active":"2024-09-11T08:24:31.121646+0000","last_peered":"2024-09-11T08:24:31.121646+0000","last_clean":"2024-09-11T08:24:31.121646+0000","last_became_active":"2024-09-11T08:24:20.606386+0000","last_became_peered":"2024-09-11T08:24:20.606386+0000","last_unstale":"2024-09-11T08:24:31.121646+0000","last_undegraded":"2024-09-11T08:24:31.121646+0000","last_fullsized":"2024-09-11T08:24:31.121646+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:28.536648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644791+0000","last_change":"2024-09-11T08:24:18.612097+0000","last_active":"2024-09-11T08:24:31.644791+0000","last_peered":"2024-09-11T08:24:31.644791+0000","last_clean":"2024-09-11T08:24:31.644791+0000","last_became_active":"2024-09-11T08:24:18.611834+0000","last_became_peered":"2024-09-11T08:24:18.611834+0000","last_unstale":"2024-09-11T08:24:31.644791+0000","last_undegraded":"2024-09-11T08:24:31.644791+0000","last_fullsized":"2024-09-11T08:24:31.644791+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:52:12.575391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643836+0000","last_change":"2024-09-11T08:24:22.628019+0000","last_active":"2024-09-11T08:24:31.643836+0000","last_peered":"2024-09-11T08:24:31.643836+0000","last_clean":"2024-09-11T08:24:31.643836+0000","last_became_active":"2024-09-11T08:24:22.627731+0000","last_became_peered":"2024-09-11T08:24:22.627731+0000","last_unstale":"2024-09-11T08:24:31.643836+0000","last_undegraded":"2024-09-11T08:24:31.643836+0000","last_fullsized":"2024-09-11T08:24:31.643836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:59:30.448604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121324+0000","last_change":"2024-09-11T08:24:24.636831+0000","last_active":"2024-09-11T08:24:31.121324+0000","last_peered":"2024-09-11T08:24:31.121324+0000","last_clean":"2024-09-11T08:24:31.121324+0000","last_became_active":"2024-09-11T08:24:24.636483+0000","last_became_peered":"2024-09-11T08:24:24.636483+0000","last_unstale":"2024-09-11T08:24:31.121324+0000","last_undegraded":"2024-09-11T08:24:31.121324+0000","last_fullsized":"2024-09-11T08:24:31.121324+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:06:41.420567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121422+0000","last_change":"2024-09-11T08:24:20.613625+0000","last_active":"2024-09-11T08:24:31.121422+0000","last_peered":"2024-09-11T08:24:31.121422+0000","last_clean":"2024-09-11T08:24:31.121422+0000","last_became_active":"2024-09-11T08:24:20.613354+0000","last_became_peered":"2024-09-11T08:24:20.613354+0000","last_unstale":"2024-09-11T08:24:31.121422+0000","last_undegraded":"2024-09-11T08:24:31.121422+0000","last_fullsized":"2024-09-11T08:24:31.121422+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:42:59.818031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"53'1","reported_seq":25,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644036+0000","last_change":"2024-09-11T08:24:18.607601+0000","last_active":"2024-09-11T08:24:31.644036+0000","last_peered":"2024-09-11T08:24:31.644036+0000","last_clean":"2024-09-11T08:24:31.644036+0000","last_became_active":"2024-09-11T08:24:18.607085+0000","last_became_peered":"2024-09-11T08:24:18.607085+0000","last_unstale":"2024-09-11T08:24:31.644036+0000","last_undegraded":"2024-09-11T08:24:31.644036+0000","last_fullsized":"2024-09-11T08:24:31.644036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:12:35.685598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690265+0000","last_change":"2024-09-11T08:24:22.619694+0000","last_active":"2024-09-11T08:25:31.690265+0000","last_peered":"2024-09-11T08:25:31.690265+0000","last_clean":"2024-09-11T08:25:31.690265+0000","last_became_active":"2024-09-11T08:24:22.619452+0000","last_became_peered":"2024-09-11T08:24:22.619452+0000","last_unstale":"2024-09-11T08:25:31.690265+0000","last_undegraded":"2024-09-11T08:25:31.690265+0000","last_fullsized":"2024-09-11T08:25:31.690265+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:55:57.467687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121552+0000","last_change":"2024-09-11T08:24:24.634484+0000","last_active":"2024-09-11T08:24:31.121552+0000","last_peered":"2024-09-11T08:24:31.121552+0000","last_clean":"2024-09-11T08:24:31.121552+0000","last_became_active":"2024-09-11T08:24:24.634134+0000","last_became_peered":"2024-09-11T08:24:24.634134+0000","last_unstale":"2024-09-11T08:24:31.121552+0000","last_undegraded":"2024-09-11T08:24:31.121552+0000","last_fullsized":"2024-09-11T08:24:31.121552+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:36:18.363418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644631+0000","last_change":"2024-09-11T08:24:20.610686+0000","last_active":"2024-09-11T08:24:31.644631+0000","last_peered":"2024-09-11T08:24:31.644631+0000","last_clean":"2024-09-11T08:24:31.644631+0000","last_became_active":"2024-09-11T08:24:20.610337+0000","last_became_peered":"2024-09-11T08:24:20.610337+0000","last_unstale":"2024-09-11T08:24:31.644631+0000","last_undegraded":"2024-09-11T08:24:31.644631+0000","last_fullsized":"2024-09-11T08:24:31.644631+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:07.158347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643904+0000","last_change":"2024-09-11T08:24:18.606710+0000","last_active":"2024-09-11T08:24:31.643904+0000","last_peered":"2024-09-11T08:24:31.643904+0000","last_clean":"2024-09-11T08:24:31.643904+0000","last_became_active":"2024-09-11T08:24:18.606432+0000","last_became_peered":"2024-09-11T08:24:18.606432+0000","last_unstale":"2024-09-11T08:24:31.643904+0000","last_undegraded":"2024-09-11T08:24:31.643904+0000","last_fullsized":"2024-09-11T08:24:31.643904+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:33:18.778917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690336+0000","last_change":"2024-09-11T08:24:22.624344+0000","last_active":"2024-09-11T08:25:31.690336+0000","last_peered":"2024-09-11T08:25:31.690336+0000","last_clean":"2024-09-11T08:25:31.690336+0000","last_became_active":"2024-09-11T08:24:22.623989+0000","last_became_peered":"2024-09-11T08:24:22.623989+0000","last_unstale":"2024-09-11T08:25:31.690336+0000","last_undegraded":"2024-09-11T08:25:31.690336+0000","last_fullsized":"2024-09-11T08:25:31.690336+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:15:36.367906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644890+0000","last_change":"2024-09-11T08:24:24.633791+0000","last_active":"2024-09-11T08:24:31.644890+0000","last_peered":"2024-09-11T08:24:31.644890+0000","last_clean":"2024-09-11T08:24:31.644890+0000","last_became_active":"2024-09-11T08:24:24.633508+0000","last_became_peered":"2024-09-11T08:24:24.633508+0000","last_unstale":"2024-09-11T08:24:31.644890+0000","last_undegraded":"2024-09-11T08:24:31.644890+0000","last_fullsized":"2024-09-11T08:24:31.644890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:40:02.496883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121677+0000","last_change":"2024-09-11T08:24:20.608269+0000","last_active":"2024-09-11T08:24:31.121677+0000","last_peered":"2024-09-11T08:24:31.121677+0000","last_clean":"2024-09-11T08:24:31.121677+0000","last_became_active":"2024-09-11T08:24:20.607905+0000","last_became_peered":"2024-09-11T08:24:20.607905+0000","last_unstale":"2024-09-11T08:24:31.121677+0000","last_undegraded":"2024-09-11T08:24:31.121677+0000","last_fullsized":"2024-09-11T08:24:31.121677+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:37:51.123445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643733+0000","last_change":"2024-09-11T08:24:18.613184+0000","last_active":"2024-09-11T08:24:31.643733+0000","last_peered":"2024-09-11T08:24:31.643733+0000","last_clean":"2024-09-11T08:24:31.643733+0000","last_became_active":"2024-09-11T08:24:18.612354+0000","last_became_peered":"2024-09-11T08:24:18.612354+0000","last_unstale":"2024-09-11T08:24:31.643733+0000","last_undegraded":"2024-09-11T08:24:31.643733+0000","last_fullsized":"2024-09-11T08:24:31.643733+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:37:35.649285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645167+0000","last_change":"2024-09-11T08:24:22.625099+0000","last_active":"2024-09-11T08:24:31.645167+0000","last_peered":"2024-09-11T08:24:31.645167+0000","last_clean":"2024-09-11T08:24:31.645167+0000","last_became_active":"2024-09-11T08:24:22.623827+0000","last_became_peered":"2024-09-11T08:24:22.623827+0000","last_unstale":"2024-09-11T08:24:31.645167+0000","last_undegraded":"2024-09-11T08:24:31.645167+0000","last_fullsized":"2024-09-11T08:24:31.645167+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:46.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306926+0000","last_change":"2024-09-11T08:24:25.336583+0000","last_active":"2024-09-11T08:24:31.306926+0000","last_peered":"2024-09-11T08:24:31.306926+0000","last_clean":"2024-09-11T08:24:31.306926+0000","last_became_active":"2024-09-11T08:24:25.336233+0000","last_became_peered":"2024-09-11T08:24:25.336233+0000","last_unstale":"2024-09-11T08:24:31.306926+0000","last_undegraded":"2024-09-11T08:24:31.306926+0000","last_fullsized":"2024-09-11T08:24:31.306926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:12:21.502321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"60'14","reported_seq":37,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121135+0000","last_change":"2024-09-11T08:24:20.608854+0000","last_active":"2024-09-11T08:24:31.121135+0000","last_peered":"2024-09-11T08:24:31.121135+0000","last_clean":"2024-09-11T08:24:31.121135+0000","last_became_active":"2024-09-11T08:24:20.608582+0000","last_became_peered":"2024-09-11T08:24:20.608582+0000","last_unstale":"2024-09-11T08:24:31.121135+0000","last_undegraded":"2024-09-11T08:24:31.121135+0000","last_fullsized":"2024-09-11T08:24:31.121135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:30:13.064049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644543+0000","last_change":"2024-09-11T08:24:18.615828+0000","last_active":"2024-09-11T08:24:31.644543+0000","last_peered":"2024-09-11T08:24:31.644543+0000","last_clean":"2024-09-11T08:24:31.644543+0000","last_became_active":"2024-09-11T08:24:18.615393+0000","last_became_peered":"2024-09-11T08:24:18.615393+0000","last_unstale":"2024-09-11T08:24:31.644543+0000","last_undegraded":"2024-09-11T08:24:31.644543+0000","last_fullsized":"2024-09-11T08:24:31.644543+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:38:00.163725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690299+0000","last_change":"2024-09-11T08:24:22.630750+0000","last_active":"2024-09-11T08:25:31.690299+0000","last_peered":"2024-09-11T08:25:31.690299+0000","last_clean":"2024-09-11T08:25:31.690299+0000","last_became_active":"2024-09-11T08:24:22.629848+0000","last_became_peered":"2024-09-11T08:24:22.629848+0000","last_unstale":"2024-09-11T08:25:31.690299+0000","last_undegraded":"2024-09-11T08:25:31.690299+0000","last_fullsized":"2024-09-11T08:25:31.690299+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:21:46.662466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644610+0000","last_change":"2024-09-11T08:24:25.338123+0000","last_active":"2024-09-11T08:24:31.644610+0000","last_peered":"2024-09-11T08:24:31.644610+0000","last_clean":"2024-09-11T08:24:31.644610+0000","last_became_active":"2024-09-11T08:24:25.337812+0000","last_became_peered":"2024-09-11T08:24:25.337812+0000","last_unstale":"2024-09-11T08:24:31.644610+0000","last_undegraded":"2024-09-11T08:24:31.644610+0000","last_fullsized":"2024-09-11T08:24:31.644610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:50:28.210131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"60'18","reported_seq":48,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306788+0000","last_change":"2024-09-11T08:24:20.979676+0000","last_active":"2024-09-11T08:24:31.306788+0000","last_peered":"2024-09-11T08:24:31.306788+0000","last_clean":"2024-09-11T08:24:31.306788+0000","last_became_active":"2024-09-11T08:24:20.979423+0000","last_became_peered":"2024-09-11T08:24:20.979423+0000","last_unstale":"2024-09-11T08:24:31.306788+0000","last_undegraded":"2024-09-11T08:24:31.306788+0000","last_fullsized":"2024-09-11T08:24:31.306788+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:57:40.839035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644430+0000","last_change":"2024-09-11T08:24:18.609802+0000","last_active":"2024-09-11T08:24:31.644430+0000","last_peered":"2024-09-11T08:24:31.644430+0000","last_clean":"2024-09-11T08:24:31.644430+0000","last_became_active":"2024-09-11T08:24:18.609496+0000","last_became_peered":"2024-09-11T08:24:18.609496+0000","last_unstale":"2024-09-11T08:24:31.644430+0000","last_undegraded":"2024-09-11T08:24:31.644430+0000","last_fullsized":"2024-09-11T08:24:31.644430+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:38:29.491263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121059+0000","last_change":"2024-09-11T08:24:22.625367+0000","last_active":"2024-09-11T08:24:31.121059+0000","last_peered":"2024-09-11T08:24:31.121059+0000","last_clean":"2024-09-11T08:24:31.121059+0000","last_became_active":"2024-09-11T08:24:22.625102+0000","last_became_peered":"2024-09-11T08:24:22.625102+0000","last_unstale":"2024-09-11T08:24:31.121059+0000","last_undegraded":"2024-09-11T08:24:31.121059+0000","last_fullsized":"2024-09-11T08:24:31.121059+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:04:50.361263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121387+0000","last_change":"2024-09-11T08:24:24.634638+0000","last_active":"2024-09-11T08:24:31.121387+0000","last_peered":"2024-09-11T08:24:31.121387+0000","last_clean":"2024-09-11T08:24:31.121387+0000","last_became_active":"2024-09-11T08:24:24.634204+0000","last_became_peered":"2024-09-11T08:24:24.634204+0000","last_unstale":"2024-09-11T08:24:31.121387+0000","last_undegraded":"2024-09-11T08:24:31.121387+0000","last_fullsized":"2024-09-11T08:24:31.121387+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:10:24.991183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"60'19","reported_seq":52,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121860+0000","last_change":"2024-09-11T08:24:20.978326+0000","last_active":"2024-09-11T08:24:31.121860+0000","last_peered":"2024-09-11T08:24:31.121860+0000","last_clean":"2024-09-11T08:24:31.121860+0000","last_became_active":"2024-09-11T08:24:20.977927+0000","last_became_peered":"2024-09-11T08:24:20.977927+0000","last_unstale":"2024-09-11T08:24:31.121860+0000","last_undegraded":"2024-09-11T08:24:31.121860+0000","last_fullsized":"2024-09-11T08:24:31.121860+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:48.746531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645630+0000","last_change":"2024-09-11T08:24:18.606686+0000","last_active":"2024-09-11T08:24:31.645630+0000","last_peered":"2024-09-11T08:24:31.645630+0000","last_clean":"2024-09-11T08:24:31.645630+0000","last_became_active":"2024-09-11T08:24:18.606039+0000","last_became_peered":"2024-09-11T08:24:18.606039+0000","last_unstale":"2024-09-11T08:24:31.645630+0000","last_undegraded":"2024-09-11T08:24:31.645630+0000","last_fullsized":"2024-09-11T08:24:31.645630+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:18:54.869573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644952+0000","last_change":"2024-09-11T08:24:22.622052+0000","last_active":"2024-09-11T08:24:31.644952+0000","last_peered":"2024-09-11T08:24:31.644952+0000","last_clean":"2024-09-11T08:24:31.644952+0000","last_became_active":"2024-09-11T08:24:22.621771+0000","last_became_peered":"2024-09-11T08:24:22.621771+0000","last_unstale":"2024-09-11T08:24:31.644952+0000","last_undegraded":"2024-09-11T08:24:31.644952+0000","last_fullsized":"2024-09-11T08:24:31.644952+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:37.670351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643612+0000","last_change":"2024-09-11T08:24:25.336629+0000","last_active":"2024-09-11T08:24:31.643612+0000","last_peered":"2024-09-11T08:24:31.643612+0000","last_clean":"2024-09-11T08:24:31.643612+0000","last_became_active":"2024-09-11T08:24:25.336246+0000","last_became_peered":"2024-09-11T08:24:25.336246+0000","last_unstale":"2024-09-11T08:24:31.643612+0000","last_undegraded":"2024-09-11T08:24:31.643612+0000","last_fullsized":"2024-09-11T08:24:31.643612+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:50:23.195512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643957+0000","last_change":"2024-09-11T08:24:20.608306+0000","last_active":"2024-09-11T08:24:31.643957+0000","last_peered":"2024-09-11T08:24:31.643957+0000","last_clean":"2024-09-11T08:24:31.643957+0000","last_became_active":"2024-09-11T08:24:20.608045+0000","last_became_peered":"2024-09-11T08:24:20.608045+0000","last_unstale":"2024-09-11T08:24:31.643957+0000","last_undegraded":"2024-09-11T08:24:31.643957+0000","last_fullsized":"2024-09-11T08:24:31.643957+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:53:50.462569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644435+0000","last_change":"2024-09-11T08:24:18.607396+0000","last_active":"2024-09-11T08:24:31.644435+0000","last_peered":"2024-09-11T08:24:31.644435+0000","last_clean":"2024-09-11T08:24:31.644435+0000","last_became_active":"2024-09-11T08:24:18.607032+0000","last_became_peered":"2024-09-11T08:24:18.607032+0000","last_unstale":"2024-09-11T08:24:31.644435+0000","last_undegraded":"2024-09-11T08:24:31.644435+0000","last_fullsized":"2024-09-11T08:24:31.644435+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:00:43.052497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690269+0000","last_change":"2024-09-11T08:24:22.622109+0000","last_active":"2024-09-11T08:25:31.690269+0000","last_peered":"2024-09-11T08:25:31.690269+0000","last_clean":"2024-09-11T08:25:31.690269+0000","last_became_active":"2024-09-11T08:24:22.621836+0000","last_became_peered":"2024-09-11T08:24:22.621836+0000","last_unstale":"2024-09-11T08:25:31.690269+0000","last_undegraded":"2024-09-11T08:25:31.690269+0000","last_fullsized":"2024-09-11T08:25:31.690269+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:01:08.166600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120905+0000","last_change":"2024-09-11T08:24:24.634138+0000","last_active":"2024-09-11T08:24:31.120905+0000","last_peered":"2024-09-11T08:24:31.120905+0000","last_clean":"2024-09-11T08:24:31.120905+0000","last_became_active":"2024-09-11T08:24:24.633855+0000","last_became_peered":"2024-09-11T08:24:24.633855+0000","last_unstale":"2024-09-11T08:24:31.120905+0000","last_undegraded":"2024-09-11T08:24:31.120905+0000","last_fullsized":"2024-09-11T08:24:31.120905+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:59:36.676165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"60'28","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690277+0000","last_change":"2024-09-11T08:24:20.613279+0000","last_active":"2024-09-11T08:25:31.690277+0000","last_peered":"2024-09-11T08:25:31.690277+0000","last_clean":"2024-09-11T08:25:31.690277+0000","last_became_active":"2024-09-11T08:24:20.612856+0000","last_became_peered":"2024-09-11T08:24:20.612856+0000","last_unstale":"2024-09-11T08:25:31.690277+0000","last_undegraded":"2024-09-11T08:25:31.690277+0000","last_fullsized":"2024-09-11T08:25:31.690277+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:57.865486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"53'1","reported_seq":26,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644933+0000","last_change":"2024-09-11T08:24:18.613258+0000","last_active":"2024-09-11T08:24:31.644933+0000","last_peered":"2024-09-11T08:24:31.644933+0000","last_clean":"2024-09-11T08:24:31.644933+0000","last_became_active":"2024-09-11T08:24:18.612980+0000","last_became_peered":"2024-09-11T08:24:18.612980+0000","last_unstale":"2024-09-11T08:24:31.644933+0000","last_undegraded":"2024-09-11T08:24:31.644933+0000","last_fullsized":"2024-09-11T08:24:31.644933+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:56:37.282029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121545+0000","last_change":"2024-09-11T08:24:22.628056+0000","last_active":"2024-09-11T08:24:31.121545+0000","last_peered":"2024-09-11T08:24:31.121545+0000","last_clean":"2024-09-11T08:24:31.121545+0000","last_became_active":"2024-09-11T08:24:22.627650+0000","last_became_peered":"2024-09-11T08:24:22.627650+0000","last_unstale":"2024-09-11T08:24:31.121545+0000","last_undegraded":"2024-09-11T08:24:31.121545+0000","last_fullsized":"2024-09-11T08:24:31.121545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:56:41.930027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307249+0000","last_change":"2024-09-11T08:24:24.629826+0000","last_active":"2024-09-11T08:24:31.307249+0000","last_peered":"2024-09-11T08:24:31.307249+0000","last_clean":"2024-09-11T08:24:31.307249+0000","last_became_active":"2024-09-11T08:24:24.629549+0000","last_became_peered":"2024-09-11T08:24:24.629549+0000","last_unstale":"2024-09-11T08:24:31.307249+0000","last_undegraded":"2024-09-11T08:24:31.307249+0000","last_fullsized":"2024-09-11T08:24:31.307249+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:46:53.802170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"60'13","reported_seq":43,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644634+0000","last_change":"2024-09-11T08:24:20.610388+0000","last_active":"2024-09-11T08:24:31.644634+0000","last_peered":"2024-09-11T08:24:31.644634+0000","last_clean":"2024-09-11T08:24:31.644634+0000","last_became_active":"2024-09-11T08:24:20.610044+0000","last_became_peered":"2024-09-11T08:24:20.610044+0000","last_unstale":"2024-09-11T08:24:31.644634+0000","last_undegraded":"2024-09-11T08:24:31.644634+0000","last_fullsized":"2024-09-11T08:24:31.644634+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:29:43.169337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"60'5","reported_seq":101,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:39.027783+0000","last_change":"2024-09-11T08:24:22.614511+0000","last_active":"2024-09-11T08:25:39.027783+0000","last_peered":"2024-09-11T08:25:39.027783+0000","last_clean":"2024-09-11T08:25:39.027783+0000","last_became_active":"2024-09-11T08:24:18.608674+0000","last_became_peered":"2024-09-11T08:24:18.608674+0000","last_unstale":"2024-09-11T08:25:39.027783+0000","last_undegraded":"2024-09-11T08:25:39.027783+0000","last_fullsized":"2024-09-11T08:25:39.027783+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:32:47.672972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013623419999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120814+0000","last_change":"2024-09-11T08:24:18.604377+0000","last_active":"2024-09-11T08:24:31.120814+0000","last_peered":"2024-09-11T08:24:31.120814+0000","last_clean":"2024-09-11T08:24:31.120814+0000","last_became_active":"2024-09-11T08:24:18.604090+0000","last_became_peered":"2024-09-11T08:24:18.604090+0000","last_unstale":"2024-09-11T08:24:31.120814+0000","last_undegraded":"2024-09-11T08:24:31.120814+0000","last_fullsized":"2024-09-11T08:24:31.120814+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:03:20.969800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121503+0000","last_change":"2024-09-11T08:24:22.620313+0000","last_active":"2024-09-11T08:24:31.121503+0000","last_peered":"2024-09-11T08:24:31.121503+0000","last_clean":"2024-09-11T08:24:31.121503+0000","last_became_active":"2024-09-11T08:24:22.620041+0000","last_became_peered":"2024-09-11T08:24:22.620041+0000","last_unstale":"2024-09-11T08:24:31.121503+0000","last_undegraded":"2024-09-11T08:24:31.121503+0000","last_fullsized":"2024-09-11T08:24:31.121503+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:55:19.652692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645432+0000","last_change":"2024-09-11T08:24:25.339645+0000","last_active":"2024-09-11T08:24:31.645432+0000","last_peered":"2024-09-11T08:24:31.645432+0000","last_clean":"2024-09-11T08:24:31.645432+0000","last_became_active":"2024-09-11T08:24:25.339388+0000","last_became_peered":"2024-09-11T08:24:25.339388+0000","last_unstale":"2024-09-11T08:24:31.645432+0000","last_undegraded":"2024-09-11T08:24:31.645432+0000","last_fullsized":"2024-09-11T08:24:31.645432+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:14:58.317284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"60'12","reported_seq":34,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122146+0000","last_change":"2024-09-11T08:24:20.609894+0000","last_active":"2024-09-11T08:24:31.122146+0000","last_peered":"2024-09-11T08:24:31.122146+0000","last_clean":"2024-09-11T08:24:31.122146+0000","last_became_active":"2024-09-11T08:24:20.609623+0000","last_became_peered":"2024-09-11T08:24:20.609623+0000","last_unstale":"2024-09-11T08:24:31.122146+0000","last_undegraded":"2024-09-11T08:24:31.122146+0000","last_fullsized":"2024-09-11T08:24:31.122146+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:17:07.507564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"55'1","reported_seq":24,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121619+0000","last_change":"2024-09-11T08:24:22.619196+0000","last_active":"2024-09-11T08:24:31.121619+0000","last_peered":"2024-09-11T08:24:31.121619+0000","last_clean":"2024-09-11T08:24:31.121619+0000","last_became_active":"2024-09-11T08:24:18.606063+0000","last_became_peered":"2024-09-11T08:24:18.606063+0000","last_unstale":"2024-09-11T08:24:31.121619+0000","last_undegraded":"2024-09-11T08:24:31.121619+0000","last_fullsized":"2024-09-11T08:24:31.121619+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:56:01.586667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038483899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643797+0000","last_change":"2024-09-11T08:24:18.613336+0000","last_active":"2024-09-11T08:24:31.643797+0000","last_peered":"2024-09-11T08:24:31.643797+0000","last_clean":"2024-09-11T08:24:31.643797+0000","last_became_active":"2024-09-11T08:24:18.612757+0000","last_became_peered":"2024-09-11T08:24:18.612757+0000","last_unstale":"2024-09-11T08:24:31.643797+0000","last_undegraded":"2024-09-11T08:24:31.643797+0000","last_fullsized":"2024-09-11T08:24:31.643797+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:06:57.231216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645203+0000","last_change":"2024-09-11T08:24:22.626749+0000","last_active":"2024-09-11T08:24:31.645203+0000","last_peered":"2024-09-11T08:24:31.645203+0000","last_clean":"2024-09-11T08:24:31.645203+0000","last_became_active":"2024-09-11T08:24:22.626490+0000","last_became_peered":"2024-09-11T08:24:22.626490+0000","last_unstale":"2024-09-11T08:24:31.645203+0000","last_undegraded":"2024-09-11T08:24:31.645203+0000","last_fullsized":"2024-09-11T08:24:31.645203+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:24.400523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643993+0000","last_change":"2024-09-11T08:24:24.629842+0000","last_active":"2024-09-11T08:24:31.643993+0000","last_peered":"2024-09-11T08:24:31.643993+0000","last_clean":"2024-09-11T08:24:31.643993+0000","last_became_active":"2024-09-11T08:24:24.629559+0000","last_became_peered":"2024-09-11T08:24:24.629559+0000","last_unstale":"2024-09-11T08:24:31.643993+0000","last_undegraded":"2024-09-11T08:24:31.643993+0000","last_fullsized":"2024-09-11T08:24:31.643993+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:19:39.030608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"60'14","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644154+0000","last_change":"2024-09-11T08:24:20.980332+0000","last_active":"2024-09-11T08:24:31.644154+0000","last_peered":"2024-09-11T08:24:31.644154+0000","last_clean":"2024-09-11T08:24:31.644154+0000","last_became_active":"2024-09-11T08:24:20.979988+0000","last_became_peered":"2024-09-11T08:24:20.979988+0000","last_unstale":"2024-09-11T08:24:31.644154+0000","last_undegraded":"2024-09-11T08:24:31.644154+0000","last_fullsized":"2024-09-11T08:24:31.644154+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:15:20.859217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"57'2","reported_seq":25,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307095+0000","last_change":"2024-09-11T08:24:22.625005+0000","last_active":"2024-09-11T08:24:31.307095+0000","last_peered":"2024-09-11T08:24:31.307095+0000","last_clean":"2024-09-11T08:24:31.307095+0000","last_became_active":"2024-09-11T08:24:18.610583+0000","last_became_peered":"2024-09-11T08:24:18.610583+0000","last_unstale":"2024-09-11T08:24:31.307095+0000","last_undegraded":"2024-09-11T08:24:31.307095+0000","last_fullsized":"2024-09-11T08:24:31.307095+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:54:03.546168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069911490000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644375+0000","last_change":"2024-09-11T08:24:18.609933+0000","last_active":"2024-09-11T08:24:31.644375+0000","last_peered":"2024-09-11T08:24:31.644375+0000","last_clean":"2024-09-11T08:24:31.644375+0000","last_became_active":"2024-09-11T08:24:18.609647+0000","last_became_peered":"2024-09-11T08:24:18.609647+0000","last_unstale":"2024-09-11T08:24:31.644375+0000","last_undegraded":"2024-09-11T08:24:31.644375+0000","last_fullsized":"2024-09-11T08:24:31.644375+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:02:39.161767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"62'79","reported_seq":52,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:24:57.026649+0000","last_change":"2024-09-11T08:24:04.836167+0000","last_active":"2024-09-11T08:24:57.026649+0000","last_peered":"2024-09-11T08:24:57.026649+0000","last_clean":"2024-09-11T08:24:57.026649+0000","last_became_active":"2024-09-11T08:24:04.830418+0000","last_became_peered":"2024-09-11T08:24:04.830418+0000","last_unstale":"2024-09-11T08:24:57.026649+0000","last_undegraded":"2024-09-11T08:24:57.026649+0000","last_fullsized":"2024-09-11T08:24:57.026649+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:22:09.006257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:22:09.006257+0000","last_clean_scrub_stamp":"2024-09-11T08:22:09.006257+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:37.204656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645044+0000","last_change":"2024-09-11T08:24:22.619513+0000","last_active":"2024-09-11T08:24:31.645044+0000","last_peered":"2024-09-11T08:24:31.645044+0000","last_clean":"2024-09-11T08:24:31.645044+0000","last_became_active":"2024-09-11T08:24:22.619040+0000","last_became_peered":"2024-09-11T08:24:22.619040+0000","last_unstale":"2024-09-11T08:24:31.645044+0000","last_undegraded":"2024-09-11T08:24:31.645044+0000","last_fullsized":"2024-09-11T08:24:31.645044+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:10.597038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645238+0000","last_change":"2024-09-11T08:24:24.632729+0000","last_active":"2024-09-11T08:24:31.645238+0000","last_peered":"2024-09-11T08:24:31.645238+0000","last_clean":"2024-09-11T08:24:31.645238+0000","last_became_active":"2024-09-11T08:24:24.632418+0000","last_became_peered":"2024-09-11T08:24:24.632418+0000","last_unstale":"2024-09-11T08:24:31.645238+0000","last_undegraded":"2024-09-11T08:24:31.645238+0000","last_fullsized":"2024-09-11T08:24:31.645238+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:15:39.234973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"60'19","reported_seq":47,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645136+0000","last_change":"2024-09-11T08:24:20.978478+0000","last_active":"2024-09-11T08:24:31.645136+0000","last_peered":"2024-09-11T08:24:31.645136+0000","last_clean":"2024-09-11T08:24:31.645136+0000","last_became_active":"2024-09-11T08:24:20.977881+0000","last_became_peered":"2024-09-11T08:24:20.977881+0000","last_unstale":"2024-09-11T08:24:31.645136+0000","last_undegraded":"2024-09-11T08:24:31.645136+0000","last_fullsized":"2024-09-11T08:24:31.645136+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:27.400531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122919+0000","last_change":"2024-09-11T08:24:18.603045+0000","last_active":"2024-09-11T08:24:31.122919+0000","last_peered":"2024-09-11T08:24:31.122919+0000","last_clean":"2024-09-11T08:24:31.122919+0000","last_became_active":"2024-09-11T08:24:18.602771+0000","last_became_peered":"2024-09-11T08:24:18.602771+0000","last_unstale":"2024-09-11T08:24:31.122919+0000","last_undegraded":"2024-09-11T08:24:31.122919+0000","last_fullsized":"2024-09-11T08:24:31.122919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:44:09.301518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120653+0000","last_change":"2024-09-11T08:24:22.623300+0000","last_active":"2024-09-11T08:24:31.120653+0000","last_peered":"2024-09-11T08:24:31.120653+0000","last_clean":"2024-09-11T08:24:31.120653+0000","last_became_active":"2024-09-11T08:24:22.623034+0000","last_became_peered":"2024-09-11T08:24:22.623034+0000","last_unstale":"2024-09-11T08:24:31.120653+0000","last_undegraded":"2024-09-11T08:24:31.120653+0000","last_fullsized":"2024-09-11T08:24:31.120653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:31.553298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643865+0000","last_change":"2024-09-11T08:24:24.635990+0000","last_active":"2024-09-11T08:24:31.643865+0000","last_peered":"2024-09-11T08:24:31.643865+0000","last_clean":"2024-09-11T08:24:31.643865+0000","last_became_active":"2024-09-11T08:24:24.635467+0000","last_became_peered":"2024-09-11T08:24:24.635467+0000","last_unstale":"2024-09-11T08:24:31.643865+0000","last_undegraded":"2024-09-11T08:24:31.643865+0000","last_fullsized":"2024-09-11T08:24:31.643865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:22:46.812249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121773+0000","last_change":"2024-09-11T08:24:20.612108+0000","last_active":"2024-09-11T08:24:31.121773+0000","last_peered":"2024-09-11T08:24:31.121773+0000","last_clean":"2024-09-11T08:24:31.121773+0000","last_became_active":"2024-09-11T08:24:20.610804+0000","last_became_peered":"2024-09-11T08:24:20.610804+0000","last_unstale":"2024-09-11T08:24:31.121773+0000","last_undegraded":"2024-09-11T08:24:31.121773+0000","last_fullsized":"2024-09-11T08:24:31.121773+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:43:42.505907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644599+0000","last_change":"2024-09-11T08:24:18.607092+0000","last_active":"2024-09-11T08:24:31.644599+0000","last_peered":"2024-09-11T08:24:31.644599+0000","last_clean":"2024-09-11T08:24:31.644599+0000","last_became_active":"2024-09-11T08:24:18.606797+0000","last_became_peered":"2024-09-11T08:24:18.606797+0000","last_unstale":"2024-09-11T08:24:31.644599+0000","last_undegraded":"2024-09-11T08:24:31.644599+0000","last_fullsized":"2024-09-11T08:24:31.644599+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:30:46.588144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121019+0000","last_change":"2024-09-11T08:24:22.627217+0000","last_active":"2024-09-11T08:24:31.121019+0000","last_peered":"2024-09-11T08:24:31.121019+0000","last_clean":"2024-09-11T08:24:31.121019+0000","last_became_active":"2024-09-11T08:24:22.626916+0000","last_became_peered":"2024-09-11T08:24:22.626916+0000","last_unstale":"2024-09-11T08:24:31.121019+0000","last_undegraded":"2024-09-11T08:24:31.121019+0000","last_fullsized":"2024-09-11T08:24:31.121019+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:32:50.766107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"60'1","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121150+0000","last_change":"2024-09-11T08:24:24.633231+0000","last_active":"2024-09-11T08:24:31.121150+0000","last_peered":"2024-09-11T08:24:31.121150+0000","last_clean":"2024-09-11T08:24:31.121150+0000","last_became_active":"2024-09-11T08:24:24.632906+0000","last_became_peered":"2024-09-11T08:24:24.632906+0000","last_unstale":"2024-09-11T08:24:31.121150+0000","last_undegraded":"2024-09-11T08:24:31.121150+0000","last_fullsized":"2024-09-11T08:24:31.121150+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:07:17.280581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643579+0000","last_change":"2024-09-11T08:24:20.979035+0000","last_active":"2024-09-11T08:24:31.643579+0000","last_peered":"2024-09-11T08:24:31.643579+0000","last_clean":"2024-09-11T08:24:31.643579+0000","last_became_active":"2024-09-11T08:24:20.978730+0000","last_became_peered":"2024-09-11T08:24:20.978730+0000","last_unstale":"2024-09-11T08:24:31.643579+0000","last_undegraded":"2024-09-11T08:24:31.643579+0000","last_fullsized":"2024-09-11T08:24:31.643579+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:15:40.965159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121681+0000","last_change":"2024-09-11T08:24:18.604560+0000","last_active":"2024-09-11T08:24:31.121681+0000","last_peered":"2024-09-11T08:24:31.121681+0000","last_clean":"2024-09-11T08:24:31.121681+0000","last_became_active":"2024-09-11T08:24:18.603758+0000","last_became_peered":"2024-09-11T08:24:18.603758+0000","last_unstale":"2024-09-11T08:24:31.121681+0000","last_undegraded":"2024-09-11T08:24:31.121681+0000","last_fullsized":"2024-09-11T08:24:31.121681+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:40:21.684231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690344+0000","last_change":"2024-09-11T08:24:22.629891+0000","last_active":"2024-09-11T08:25:31.690344+0000","last_peered":"2024-09-11T08:25:31.690344+0000","last_clean":"2024-09-11T08:25:31.690344+0000","last_became_active":"2024-09-11T08:24:22.629368+0000","last_became_peered":"2024-09-11T08:24:22.629368+0000","last_unstale":"2024-09-11T08:25:31.690344+0000","last_undegraded":"2024-09-11T08:25:31.690344+0000","last_fullsized":"2024-09-11T08:25:31.690344+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:17:33.583486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644581+0000","last_change":"2024-09-11T08:24:25.336555+0000","last_active":"2024-09-11T08:24:31.644581+0000","last_peered":"2024-09-11T08:24:31.644581+0000","last_clean":"2024-09-11T08:24:31.644581+0000","last_became_active":"2024-09-11T08:24:25.336253+0000","last_became_peered":"2024-09-11T08:24:25.336253+0000","last_unstale":"2024-09-11T08:24:31.644581+0000","last_undegraded":"2024-09-11T08:24:31.644581+0000","last_fullsized":"2024-09-11T08:24:31.644581+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:47:38.003757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"60'12","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121094+0000","last_change":"2024-09-11T08:24:20.614181+0000","last_active":"2024-09-11T08:24:31.121094+0000","last_peered":"2024-09-11T08:24:31.121094+0000","last_clean":"2024-09-11T08:24:31.121094+0000","last_became_active":"2024-09-11T08:24:20.613848+0000","last_became_peered":"2024-09-11T08:24:20.613848+0000","last_unstale":"2024-09-11T08:24:31.121094+0000","last_undegraded":"2024-09-11T08:24:31.121094+0000","last_fullsized":"2024-09-11T08:24:31.121094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:50:38.254311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"53'2","reported_seq":34,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121910+0000","last_change":"2024-09-11T08:24:18.612162+0000","last_active":"2024-09-11T08:24:31.121910+0000","last_peered":"2024-09-11T08:24:31.121910+0000","last_clean":"2024-09-11T08:24:31.121910+0000","last_became_active":"2024-09-11T08:24:18.611899+0000","last_became_peered":"2024-09-11T08:24:18.611899+0000","last_unstale":"2024-09-11T08:24:31.121910+0000","last_undegraded":"2024-09-11T08:24:31.121910+0000","last_fullsized":"2024-09-11T08:24:31.121910+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:17.821188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120559+0000","last_change":"2024-09-11T08:24:22.616809+0000","last_active":"2024-09-11T08:24:31.120559+0000","last_peered":"2024-09-11T08:24:31.120559+0000","last_clean":"2024-09-11T08:24:31.120559+0000","last_became_active":"2024-09-11T08:24:22.616482+0000","last_became_peered":"2024-09-11T08:24:22.616482+0000","last_unstale":"2024-09-11T08:24:31.120559+0000","last_undegraded":"2024-09-11T08:24:31.120559+0000","last_fullsized":"2024-09-11T08:24:31.120559+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:14:34.651713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306890+0000","last_change":"2024-09-11T08:24:24.629705+0000","last_active":"2024-09-11T08:24:31.306890+0000","last_peered":"2024-09-11T08:24:31.306890+0000","last_clean":"2024-09-11T08:24:31.306890+0000","last_became_active":"2024-09-11T08:24:24.629423+0000","last_became_peered":"2024-09-11T08:24:24.629423+0000","last_unstale":"2024-09-11T08:24:31.306890+0000","last_undegraded":"2024-09-11T08:24:31.306890+0000","last_fullsized":"2024-09-11T08:24:31.306890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:41:33.829844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122087+0000","last_change":"2024-09-11T08:24:20.978644+0000","last_active":"2024-09-11T08:24:31.122087+0000","last_peered":"2024-09-11T08:24:31.122087+0000","last_clean":"2024-09-11T08:24:31.122087+0000","last_became_active":"2024-09-11T08:24:20.978167+0000","last_became_peered":"2024-09-11T08:24:20.978167+0000","last_unstale":"2024-09-11T08:24:31.122087+0000","last_undegraded":"2024-09-11T08:24:31.122087+0000","last_fullsized":"2024-09-11T08:24:31.122087+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:23.312183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120708+0000","last_change":"2024-09-11T08:24:18.606219+0000","last_active":"2024-09-11T08:24:31.120708+0000","last_peered":"2024-09-11T08:24:31.120708+0000","last_clean":"2024-09-11T08:24:31.120708+0000","last_became_active":"2024-09-11T08:24:18.605954+0000","last_became_peered":"2024-09-11T08:24:18.605954+0000","last_unstale":"2024-09-11T08:24:31.120708+0000","last_undegraded":"2024-09-11T08:24:31.120708+0000","last_fullsized":"2024-09-11T08:24:31.120708+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:21:04.850766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307449+0000","last_change":"2024-09-11T08:24:22.626701+0000","last_active":"2024-09-11T08:24:31.307449+0000","last_peered":"2024-09-11T08:24:31.307449+0000","last_clean":"2024-09-11T08:24:31.307449+0000","last_became_active":"2024-09-11T08:24:22.626351+0000","last_became_peered":"2024-09-11T08:24:22.626351+0000","last_unstale":"2024-09-11T08:24:31.307449+0000","last_undegraded":"2024-09-11T08:24:31.307449+0000","last_fullsized":"2024-09-11T08:24:31.307449+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:01:44.427374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122010+0000","last_change":"2024-09-11T08:24:24.638035+0000","last_active":"2024-09-11T08:24:31.122010+0000","last_peered":"2024-09-11T08:24:31.122010+0000","last_clean":"2024-09-11T08:24:31.122010+0000","last_became_active":"2024-09-11T08:24:24.637835+0000","last_became_peered":"2024-09-11T08:24:24.637835+0000","last_unstale":"2024-09-11T08:24:31.122010+0000","last_undegraded":"2024-09-11T08:24:31.122010+0000","last_fullsized":"2024-09-11T08:24:31.122010+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:47:54.013004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644045+0000","last_change":"2024-09-11T08:24:18.607733+0000","last_active":"2024-09-11T08:24:31.644045+0000","last_peered":"2024-09-11T08:24:31.644045+0000","last_clean":"2024-09-11T08:24:31.644045+0000","last_became_active":"2024-09-11T08:24:18.607435+0000","last_became_peered":"2024-09-11T08:24:18.607435+0000","last_unstale":"2024-09-11T08:24:31.644045+0000","last_undegraded":"2024-09-11T08:24:31.644045+0000","last_fullsized":"2024-09-11T08:24:31.644045+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:37:46.670160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"60'6","reported_seq":25,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307560+0000","last_change":"2024-09-11T08:24:20.979607+0000","last_active":"2024-09-11T08:24:31.307560+0000","last_peered":"2024-09-11T08:24:31.307560+0000","last_clean":"2024-09-11T08:24:31.307560+0000","last_became_active":"2024-09-11T08:24:20.979361+0000","last_became_peered":"2024-09-11T08:24:20.979361+0000","last_unstale":"2024-09-11T08:24:31.307560+0000","last_undegraded":"2024-09-11T08:24:31.307560+0000","last_fullsized":"2024-09-11T08:24:31.307560+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:01.257490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644976+0000","last_change":"2024-09-11T08:24:22.625848+0000","last_active":"2024-09-11T08:24:31.644976+0000","last_peered":"2024-09-11T08:24:31.644976+0000","last_clean":"2024-09-11T08:24:31.644976+0000","last_became_active":"2024-09-11T08:24:22.625511+0000","last_became_peered":"2024-09-11T08:24:22.625511+0000","last_unstale":"2024-09-11T08:24:31.644976+0000","last_undegraded":"2024-09-11T08:24:31.644976+0000","last_fullsized":"2024-09-11T08:24:31.644976+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:12:48.901989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645340+0000","last_change":"2024-09-11T08:24:25.338198+0000","last_active":"2024-09-11T08:24:31.645340+0000","last_peered":"2024-09-11T08:24:31.645340+0000","last_clean":"2024-09-11T08:24:31.645340+0000","last_became_active":"2024-09-11T08:24:25.337825+0000","last_became_peered":"2024-09-11T08:24:25.337825+0000","last_unstale":"2024-09-11T08:24:31.645340+0000","last_undegraded":"2024-09-11T08:24:31.645340+0000","last_fullsized":"2024-09-11T08:24:31.645340+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:35:13.673933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307479+0000","last_change":"2024-09-11T08:24:20.978174+0000","last_active":"2024-09-11T08:24:31.307479+0000","last_peered":"2024-09-11T08:24:31.307479+0000","last_clean":"2024-09-11T08:24:31.307479+0000","last_became_active":"2024-09-11T08:24:20.977918+0000","last_became_peered":"2024-09-11T08:24:20.977918+0000","last_unstale":"2024-09-11T08:24:31.307479+0000","last_undegraded":"2024-09-11T08:24:31.307479+0000","last_fullsized":"2024-09-11T08:24:31.307479+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:43.472987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644837+0000","last_change":"2024-09-11T08:24:18.606855+0000","last_active":"2024-09-11T08:24:31.644837+0000","last_peered":"2024-09-11T08:24:31.644837+0000","last_clean":"2024-09-11T08:24:31.644837+0000","last_became_active":"2024-09-11T08:24:18.606502+0000","last_became_peered":"2024-09-11T08:24:18.606502+0000","last_unstale":"2024-09-11T08:24:31.644837+0000","last_undegraded":"2024-09-11T08:24:31.644837+0000","last_fullsized":"2024-09-11T08:24:31.644837+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:48:30.487464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"60'8","reported_seq":38,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690339+0000","last_change":"2024-09-11T08:24:22.623958+0000","last_active":"2024-09-11T08:25:31.690339+0000","last_peered":"2024-09-11T08:25:31.690339+0000","last_clean":"2024-09-11T08:25:31.690339+0000","last_became_active":"2024-09-11T08:24:22.623620+0000","last_became_peered":"2024-09-11T08:24:22.623620+0000","last_unstale":"2024-09-11T08:25:31.690339+0000","last_undegraded":"2024-09-11T08:25:31.690339+0000","last_fullsized":"2024-09-11T08:25:31.690339+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:51:10.471040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121597+0000","last_change":"2024-09-11T08:24:24.633210+0000","last_active":"2024-09-11T08:24:31.121597+0000","last_peered":"2024-09-11T08:24:31.121597+0000","last_clean":"2024-09-11T08:24:31.121597+0000","last_became_active":"2024-09-11T08:24:24.632820+0000","last_became_peered":"2024-09-11T08:24:24.632820+0000","last_unstale":"2024-09-11T08:24:31.121597+0000","last_undegraded":"2024-09-11T08:24:31.121597+0000","last_fullsized":"2024-09-11T08:24:31.121597+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:29:48.776181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645101+0000","last_change":"2024-09-11T08:24:20.978176+0000","last_active":"2024-09-11T08:24:31.645101+0000","last_peered":"2024-09-11T08:24:31.645101+0000","last_clean":"2024-09-11T08:24:31.645101+0000","last_became_active":"2024-09-11T08:24:20.977887+0000","last_became_peered":"2024-09-11T08:24:20.977887+0000","last_unstale":"2024-09-11T08:24:31.645101+0000","last_undegraded":"2024-09-11T08:24:31.645101+0000","last_fullsized":"2024-09-11T08:24:31.645101+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:54:35.258534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121940+0000","last_change":"2024-09-11T08:24:18.606550+0000","last_active":"2024-09-11T08:24:31.121940+0000","last_peered":"2024-09-11T08:24:31.121940+0000","last_clean":"2024-09-11T08:24:31.121940+0000","last_became_active":"2024-09-11T08:24:18.606274+0000","last_became_peered":"2024-09-11T08:24:18.606274+0000","last_unstale":"2024-09-11T08:24:31.121940+0000","last_undegraded":"2024-09-11T08:24:31.121940+0000","last_fullsized":"2024-09-11T08:24:31.121940+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:15:44.540152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"60'8","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690426+0000","last_change":"2024-09-11T08:24:22.626647+0000","last_active":"2024-09-11T08:25:31.690426+0000","last_peered":"2024-09-11T08:25:31.690426+0000","last_clean":"2024-09-11T08:25:31.690426+0000","last_became_active":"2024-09-11T08:24:22.626296+0000","last_became_peered":"2024-09-11T08:24:22.626296+0000","last_unstale":"2024-09-11T08:25:31.690426+0000","last_undegraded":"2024-09-11T08:25:31.690426+0000","last_fullsized":"2024-09-11T08:25:31.690426+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:45:26.487207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121806+0000","last_change":"2024-09-11T08:24:24.634883+0000","last_active":"2024-09-11T08:24:31.121806+0000","last_peered":"2024-09-11T08:24:31.121806+0000","last_clean":"2024-09-11T08:24:31.121806+0000","last_became_active":"2024-09-11T08:24:24.634571+0000","last_became_peered":"2024-09-11T08:24:24.634571+0000","last_unstale":"2024-09-11T08:24:31.121806+0000","last_undegraded":"2024-09-11T08:24:31.121806+0000","last_fullsized":"2024-09-11T08:24:31.121806+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:56:03.091789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"60'9","reported_seq":32,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643824+0000","last_change":"2024-09-11T08:24:20.608916+0000","last_active":"2024-09-11T08:24:31.643824+0000","last_peered":"2024-09-11T08:24:31.643824+0000","last_clean":"2024-09-11T08:24:31.643824+0000","last_became_active":"2024-09-11T08:24:20.608642+0000","last_became_peered":"2024-09-11T08:24:20.608642+0000","last_unstale":"2024-09-11T08:24:31.643824+0000","last_undegraded":"2024-09-11T08:24:31.643824+0000","last_fullsized":"2024-09-11T08:24:31.643824+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:24:58.368361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644068+0000","last_change":"2024-09-11T08:24:18.610475+0000","last_active":"2024-09-11T08:24:31.644068+0000","last_peered":"2024-09-11T08:24:31.644068+0000","last_clean":"2024-09-11T08:24:31.644068+0000","last_became_active":"2024-09-11T08:24:18.610205+0000","last_became_peered":"2024-09-11T08:24:18.610205+0000","last_unstale":"2024-09-11T08:24:31.644068+0000","last_undegraded":"2024-09-11T08:24:31.644068+0000","last_fullsized":"2024-09-11T08:24:31.644068+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:44:05.787800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307014+0000","last_change":"2024-09-11T08:24:22.630585+0000","last_active":"2024-09-11T08:24:31.307014+0000","last_peered":"2024-09-11T08:24:31.307014+0000","last_clean":"2024-09-11T08:24:31.307014+0000","last_became_active":"2024-09-11T08:24:22.630316+0000","last_became_peered":"2024-09-11T08:24:22.630316+0000","last_unstale":"2024-09-11T08:24:31.307014+0000","last_undegraded":"2024-09-11T08:24:31.307014+0000","last_fullsized":"2024-09-11T08:24:31.307014+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:08:13.012767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.120812+0000","last_change":"2024-09-11T08:24:24.632120+0000","last_active":"2024-09-11T08:24:31.120812+0000","last_peered":"2024-09-11T08:24:31.120812+0000","last_clean":"2024-09-11T08:24:31.120812+0000","last_became_active":"2024-09-11T08:24:24.631620+0000","last_became_peered":"2024-09-11T08:24:24.631620+0000","last_unstale":"2024-09-11T08:24:31.120812+0000","last_undegraded":"2024-09-11T08:24:31.120812+0000","last_fullsized":"2024-09-11T08:24:31.120812+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:31:39.994685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121202+0000","last_change":"2024-09-11T08:24:20.606969+0000","last_active":"2024-09-11T08:24:31.121202+0000","last_peered":"2024-09-11T08:24:31.121202+0000","last_clean":"2024-09-11T08:24:31.121202+0000","last_became_active":"2024-09-11T08:24:20.606406+0000","last_became_peered":"2024-09-11T08:24:20.606406+0000","last_unstale":"2024-09-11T08:24:31.121202+0000","last_undegraded":"2024-09-11T08:24:31.121202+0000","last_fullsized":"2024-09-11T08:24:31.121202+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:01:26.826903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644643+0000","last_change":"2024-09-11T08:24:18.607479+0000","last_active":"2024-09-11T08:24:31.644643+0000","last_peered":"2024-09-11T08:24:31.644643+0000","last_clean":"2024-09-11T08:24:31.644643+0000","last_became_active":"2024-09-11T08:24:18.606062+0000","last_became_peered":"2024-09-11T08:24:18.606062+0000","last_unstale":"2024-09-11T08:24:31.644643+0000","last_undegraded":"2024-09-11T08:24:31.644643+0000","last_fullsized":"2024-09-11T08:24:31.644643+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:07:38.453373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643734+0000","last_change":"2024-09-11T08:24:22.625180+0000","last_active":"2024-09-11T08:24:31.643734+0000","last_peered":"2024-09-11T08:24:31.643734+0000","last_clean":"2024-09-11T08:24:31.643734+0000","last_became_active":"2024-09-11T08:24:22.624879+0000","last_became_peered":"2024-09-11T08:24:22.624879+0000","last_unstale":"2024-09-11T08:24:31.643734+0000","last_undegraded":"2024-09-11T08:24:31.643734+0000","last_fullsized":"2024-09-11T08:24:31.643734+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:51:13.770008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306586+0000","last_change":"2024-09-11T08:24:24.635962+0000","last_active":"2024-09-11T08:24:31.306586+0000","last_peered":"2024-09-11T08:24:31.306586+0000","last_clean":"2024-09-11T08:24:31.306586+0000","last_became_active":"2024-09-11T08:24:24.635651+0000","last_became_peered":"2024-09-11T08:24:24.635651+0000","last_unstale":"2024-09-11T08:24:31.306586+0000","last_undegraded":"2024-09-11T08:24:31.306586+0000","last_fullsized":"2024-09-11T08:24:31.306586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:12:07.881390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"60'4","reported_seq":22,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306586+0000","last_change":"2024-09-11T08:24:20.614336+0000","last_active":"2024-09-11T08:24:31.306586+0000","last_peered":"2024-09-11T08:24:31.306586+0000","last_clean":"2024-09-11T08:24:31.306586+0000","last_became_active":"2024-09-11T08:24:20.614082+0000","last_became_peered":"2024-09-11T08:24:20.614082+0000","last_unstale":"2024-09-11T08:24:31.306586+0000","last_undegraded":"2024-09-11T08:24:31.306586+0000","last_fullsized":"2024-09-11T08:24:31.306586+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:52:48.491779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645103+0000","last_change":"2024-09-11T08:24:18.610784+0000","last_active":"2024-09-11T08:24:31.645103+0000","last_peered":"2024-09-11T08:24:31.645103+0000","last_clean":"2024-09-11T08:24:31.645103+0000","last_became_active":"2024-09-11T08:24:18.610524+0000","last_became_peered":"2024-09-11T08:24:18.610524+0000","last_unstale":"2024-09-11T08:24:31.645103+0000","last_undegraded":"2024-09-11T08:24:31.645103+0000","last_fullsized":"2024-09-11T08:24:31.645103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:26:18.290089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643853+0000","last_change":"2024-09-11T08:24:22.625145+0000","last_active":"2024-09-11T08:24:31.643853+0000","last_peered":"2024-09-11T08:24:31.643853+0000","last_clean":"2024-09-11T08:24:31.643853+0000","last_became_active":"2024-09-11T08:24:22.624888+0000","last_became_peered":"2024-09-11T08:24:22.624888+0000","last_unstale":"2024-09-11T08:24:31.643853+0000","last_undegraded":"2024-09-11T08:24:31.643853+0000","last_fullsized":"2024-09-11T08:24:31.643853+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:11:19.576615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644112+0000","last_change":"2024-09-11T08:24:24.635266+0000","last_active":"2024-09-11T08:24:31.644112+0000","last_peered":"2024-09-11T08:24:31.644112+0000","last_clean":"2024-09-11T08:24:31.644112+0000","last_became_active":"2024-09-11T08:24:24.634935+0000","last_became_peered":"2024-09-11T08:24:24.634935+0000","last_unstale":"2024-09-11T08:24:31.644112+0000","last_undegraded":"2024-09-11T08:24:31.644112+0000","last_fullsized":"2024-09-11T08:24:31.644112+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:58:05.175649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306762+0000","last_change":"2024-09-11T08:24:20.978167+0000","last_active":"2024-09-11T08:24:31.306762+0000","last_peered":"2024-09-11T08:24:31.306762+0000","last_clean":"2024-09-11T08:24:31.306762+0000","last_became_active":"2024-09-11T08:24:20.977877+0000","last_became_peered":"2024-09-11T08:24:20.977877+0000","last_unstale":"2024-09-11T08:24:31.306762+0000","last_undegraded":"2024-09-11T08:24:31.306762+0000","last_fullsized":"2024-09-11T08:24:31.306762+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:39:42.639634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644469+0000","last_change":"2024-09-11T08:24:18.614076+0000","last_active":"2024-09-11T08:24:31.644469+0000","last_peered":"2024-09-11T08:24:31.644469+0000","last_clean":"2024-09-11T08:24:31.644469+0000","last_became_active":"2024-09-11T08:24:18.613789+0000","last_became_peered":"2024-09-11T08:24:18.613789+0000","last_unstale":"2024-09-11T08:24:31.644469+0000","last_undegraded":"2024-09-11T08:24:31.644469+0000","last_fullsized":"2024-09-11T08:24:31.644469+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:39:35.120681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643479+0000","last_change":"2024-09-11T08:24:22.628532+0000","last_active":"2024-09-11T08:24:31.643479+0000","last_peered":"2024-09-11T08:24:31.643479+0000","last_clean":"2024-09-11T08:24:31.643479+0000","last_became_active":"2024-09-11T08:24:22.628211+0000","last_became_peered":"2024-09-11T08:24:22.628211+0000","last_unstale":"2024-09-11T08:24:31.643479+0000","last_undegraded":"2024-09-11T08:24:31.643479+0000","last_fullsized":"2024-09-11T08:24:31.643479+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:01:14.610567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.306746+0000","last_change":"2024-09-11T08:24:25.338172+0000","last_active":"2024-09-11T08:24:31.306746+0000","last_peered":"2024-09-11T08:24:31.306746+0000","last_clean":"2024-09-11T08:24:31.306746+0000","last_became_active":"2024-09-11T08:24:25.337797+0000","last_became_peered":"2024-09-11T08:24:25.337797+0000","last_unstale":"2024-09-11T08:24:31.306746+0000","last_undegraded":"2024-09-11T08:24:31.306746+0000","last_fullsized":"2024-09-11T08:24:31.306746+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:35:20.444353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"60'1","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644966+0000","last_change":"2024-09-11T08:24:24.635456+0000","last_active":"2024-09-11T08:24:31.644966+0000","last_peered":"2024-09-11T08:24:31.644966+0000","last_clean":"2024-09-11T08:24:31.644966+0000","last_became_active":"2024-09-11T08:24:24.635167+0000","last_became_peered":"2024-09-11T08:24:24.635167+0000","last_unstale":"2024-09-11T08:24:31.644966+0000","last_undegraded":"2024-09-11T08:24:31.644966+0000","last_fullsized":"2024-09-11T08:24:31.644966+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:19:15.885301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.643653+0000","last_change":"2024-09-11T08:24:18.605425+0000","last_active":"2024-09-11T08:24:31.643653+0000","last_peered":"2024-09-11T08:24:31.643653+0000","last_clean":"2024-09-11T08:24:31.643653+0000","last_became_active":"2024-09-11T08:24:18.604978+0000","last_became_peered":"2024-09-11T08:24:18.604978+0000","last_unstale":"2024-09-11T08:24:31.643653+0000","last_undegraded":"2024-09-11T08:24:31.643653+0000","last_fullsized":"2024-09-11T08:24:31.643653+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:03:33.609832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"60'10","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122019+0000","last_change":"2024-09-11T08:24:20.980359+0000","last_active":"2024-09-11T08:24:31.122019+0000","last_peered":"2024-09-11T08:24:31.122019+0000","last_clean":"2024-09-11T08:24:31.122019+0000","last_became_active":"2024-09-11T08:24:20.980095+0000","last_became_peered":"2024-09-11T08:24:20.980095+0000","last_unstale":"2024-09-11T08:24:31.122019+0000","last_undegraded":"2024-09-11T08:24:31.122019+0000","last_fullsized":"2024-09-11T08:24:31.122019+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:52:00.254471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"60'8","reported_seq":41,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-11T08:25:31.690377+0000","last_change":"2024-09-11T08:24:22.625357+0000","last_active":"2024-09-11T08:25:31.690377+0000","last_peered":"2024-09-11T08:25:31.690377+0000","last_clean":"2024-09-11T08:25:31.690377+0000","last_became_active":"2024-09-11T08:24:22.623565+0000","last_became_peered":"2024-09-11T08:24:22.623565+0000","last_unstale":"2024-09-11T08:25:31.690377+0000","last_undegraded":"2024-09-11T08:25:31.690377+0000","last_fullsized":"2024-09-11T08:25:31.690377+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:45:24.474162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644038+0000","last_change":"2024-09-11T08:24:24.630905+0000","last_active":"2024-09-11T08:24:31.644038+0000","last_peered":"2024-09-11T08:24:31.644038+0000","last_clean":"2024-09-11T08:24:31.644038+0000","last_became_active":"2024-09-11T08:24:24.630613+0000","last_became_peered":"2024-09-11T08:24:24.630613+0000","last_unstale":"2024-09-11T08:24:31.644038+0000","last_undegraded":"2024-09-11T08:24:31.644038+0000","last_fullsized":"2024-09-11T08:24:31.644038+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:15:02.340916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307036+0000","last_change":"2024-09-11T08:24:18.609433+0000","last_active":"2024-09-11T08:24:31.307036+0000","last_peered":"2024-09-11T08:24:31.307036+0000","last_clean":"2024-09-11T08:24:31.307036+0000","last_became_active":"2024-09-11T08:24:18.608650+0000","last_became_peered":"2024-09-11T08:24:18.608650+0000","last_unstale":"2024-09-11T08:24:31.307036+0000","last_undegraded":"2024-09-11T08:24:31.307036+0000","last_fullsized":"2024-09-11T08:24:31.307036+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:31:13.399425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"60'11","reported_seq":35,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644392+0000","last_change":"2024-09-11T08:24:20.611153+0000","last_active":"2024-09-11T08:24:31.644392+0000","last_peered":"2024-09-11T08:24:31.644392+0000","last_clean":"2024-09-11T08:24:31.644392+0000","last_became_active":"2024-09-11T08:24:20.610802+0000","last_became_peered":"2024-09-11T08:24:20.610802+0000","last_unstale":"2024-09-11T08:24:31.644392+0000","last_undegraded":"2024-09-11T08:24:31.644392+0000","last_fullsized":"2024-09-11T08:24:31.644392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:47:48.881371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122024+0000","last_change":"2024-09-11T08:24:22.623152+0000","last_active":"2024-09-11T08:24:31.122024+0000","last_peered":"2024-09-11T08:24:31.122024+0000","last_clean":"2024-09-11T08:24:31.122024+0000","last_became_active":"2024-09-11T08:24:22.622845+0000","last_became_peered":"2024-09-11T08:24:22.622845+0000","last_unstale":"2024-09-11T08:24:31.122024+0000","last_undegraded":"2024-09-11T08:24:31.122024+0000","last_fullsized":"2024-09-11T08:24:31.122024+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:57:28.315295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121603+0000","last_change":"2024-09-11T08:24:25.338725+0000","last_active":"2024-09-11T08:24:31.121603+0000","last_peered":"2024-09-11T08:24:31.121603+0000","last_clean":"2024-09-11T08:24:31.121603+0000","last_became_active":"2024-09-11T08:24:25.338450+0000","last_became_peered":"2024-09-11T08:24:25.338450+0000","last_unstale":"2024-09-11T08:24:31.121603+0000","last_undegraded":"2024-09-11T08:24:31.121603+0000","last_fullsized":"2024-09-11T08:24:31.121603+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:22:27.532897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.645215+0000","last_change":"2024-09-11T08:24:18.609876+0000","last_active":"2024-09-11T08:24:31.645215+0000","last_peered":"2024-09-11T08:24:31.645215+0000","last_clean":"2024-09-11T08:24:31.645215+0000","last_became_active":"2024-09-11T08:24:18.608554+0000","last_became_peered":"2024-09-11T08:24:18.608554+0000","last_unstale":"2024-09-11T08:24:31.645215+0000","last_undegraded":"2024-09-11T08:24:31.645215+0000","last_fullsized":"2024-09-11T08:24:31.645215+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:06:20.895951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"60'15","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.122255+0000","last_change":"2024-09-11T08:24:20.613552+0000","last_active":"2024-09-11T08:24:31.122255+0000","last_peered":"2024-09-11T08:24:31.122255+0000","last_clean":"2024-09-11T08:24:31.122255+0000","last_became_active":"2024-09-11T08:24:20.613263+0000","last_became_peered":"2024-09-11T08:24:20.613263+0000","last_unstale":"2024-09-11T08:24:31.122255+0000","last_undegraded":"2024-09-11T08:24:31.122255+0000","last_fullsized":"2024-09-11T08:24:31.122255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:06:27.820868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.644232+0000","last_change":"2024-09-11T08:24:22.629394+0000","last_active":"2024-09-11T08:24:31.644232+0000","last_peered":"2024-09-11T08:24:31.644232+0000","last_clean":"2024-09-11T08:24:31.644232+0000","last_became_active":"2024-09-11T08:24:22.629114+0000","last_became_peered":"2024-09-11T08:24:22.629114+0000","last_unstale":"2024-09-11T08:24:31.644232+0000","last_undegraded":"2024-09-11T08:24:31.644232+0000","last_fullsized":"2024-09-11T08:24:31.644232+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:57:42.765458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307488+0000","last_change":"2024-09-11T08:24:25.336832+0000","last_active":"2024-09-11T08:24:31.307488+0000","last_peered":"2024-09-11T08:24:31.307488+0000","last_clean":"2024-09-11T08:24:31.307488+0000","last_became_active":"2024-09-11T08:24:25.336466+0000","last_became_peered":"2024-09-11T08:24:25.336466+0000","last_unstale":"2024-09-11T08:24:31.307488+0000","last_undegraded":"2024-09-11T08:24:31.307488+0000","last_fullsized":"2024-09-11T08:24:31.307488+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:23.609427+0000","last_clean_scrub_stamp":"2024-09-11T08:24:23.609427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:35.627909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307148+0000","last_change":"2024-09-11T08:24:18.614611+0000","last_active":"2024-09-11T08:24:31.307148+0000","last_peered":"2024-09-11T08:24:31.307148+0000","last_clean":"2024-09-11T08:24:31.307148+0000","last_became_active":"2024-09-11T08:24:18.614295+0000","last_became_peered":"2024-09-11T08:24:18.614295+0000","last_unstale":"2024-09-11T08:24:31.307148+0000","last_undegraded":"2024-09-11T08:24:31.307148+0000","last_fullsized":"2024-09-11T08:24:31.307148+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:17.592695+0000","last_clean_scrub_stamp":"2024-09-11T08:24:17.592695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:10:21.065116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"60'12","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.307206+0000","last_change":"2024-09-11T08:24:20.616441+0000","last_active":"2024-09-11T08:24:31.307206+0000","last_peered":"2024-09-11T08:24:31.307206+0000","last_clean":"2024-09-11T08:24:31.307206+0000","last_became_active":"2024-09-11T08:24:20.616151+0000","last_became_peered":"2024-09-11T08:24:20.616151+0000","last_unstale":"2024-09-11T08:24:31.307206+0000","last_undegraded":"2024-09-11T08:24:31.307206+0000","last_fullsized":"2024-09-11T08:24:31.307206+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:19.595621+0000","last_clean_scrub_stamp":"2024-09-11T08:24:19.595621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:39.015560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-11T08:24:31.121411+0000","last_change":"2024-09-11T08:24:22.627742+0000","last_active":"2024-09-11T08:24:31.121411+0000","last_peered":"2024-09-11T08:24:31.121411+0000","last_clean":"2024-09-11T08:24:31.121411+0000","last_became_active":"2024-09-11T08:24:22.627089+0000","last_became_peered":"2024-09-11T08:24:22.627089+0000","last_unstale":"2024-09-11T08:24:31.121411+0000","last_undegraded":"2024-09-11T08:24:31.121411+0000","last_fullsized":"2024-09-11T08:24:31.121411+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:24:21.599548+0000","last_clean_scrub_stamp":"2024-09-11T08:24:21.599548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:28:46.573033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397526,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298548,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446604,"statfs":{"total":95995035648,"available":95689322496,"internally_reserved":0,"allocated":1220608,"data_stored":825318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.44400000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.47199999999999998}]},{"osd":2,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.39800000000000002}]},{"osd":3,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.379}]},{"osd":4,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.315},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.33200000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.40999999999999998}]},{"osd":6,"last update":"Wed Sep 11 08:25:03 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.38600000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593756,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298524,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446628,"statfs":{"total":95995035648,"available":95689347072,"internally_reserved":0,"allocated":1196032,"data_stored":824985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.377},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.42599999999999999}]},{"osd":2,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.41199999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.371},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.30399999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.255},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.28199999999999997}]},{"osd":5,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.318},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32900000000000001}]},{"osd":7,"last update":"Wed Sep 11 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.36199999999999999}]}]},{"osd":5,"up_from":37,"seq":158913789982,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297944,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447208,"statfs":{"total":95995035648,"available":95689940992,"internally_reserved":0,"allocated":602112,"data_stored":234532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73299999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52500000000000002,"15min":0.53300000000000003},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.63900000000000001,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":6.7130000000000001}]},{"osd":1,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.68300000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54100000000000004,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70199999999999996,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.40899999999999997}]},{"osd":2,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52400000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.65200000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":6.7290000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.56399999999999995,"15min":0.57299999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71099999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.34599999999999997}]},{"osd":3,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":6.7690000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67000000000000004,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.48199999999999998}]},{"osd":4,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.375,"15min":0.36699999999999999},"min":{"1min":0.28799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66600000000000004,"5min":0.874,"15min":0.874},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.36599999999999999,"15min":0.35799999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38500000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.502},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.47199999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.44},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.44900000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297964,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447188,"statfs":{"total":95995035648,"available":95689920512,"internally_reserved":0,"allocated":622592,"data_stored":234426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.79400000000000004,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.441},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69199999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36699999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.45200000000000001}]},{"osd":2,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.376},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70799999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.42399999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.91000000000000003,"15min":0.96999999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69099999999999995,"5min":11.728999999999999,"15min":11.728999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.36899999999999999,"15min":0.35699999999999998},"min":{"1min":0.27600000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.311},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.40000000000000002,"15min":0.38700000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.28199999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.35399999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297984,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447168,"statfs":{"total":95995035648,"available":95689900032,"internally_reserved":0,"allocated":643072,"data_stored":235139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.36599999999999999,"15min":0.375},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.42399999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.372,"15min":0.38},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.499,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Wed Sep 11 08:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.40000000000000002,"15min":0.41299999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.48999999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.33100000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.48799999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.315},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.40400000000000003,"15min":0.40999999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.32300000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.67800000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68899999999999995,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.77600000000000002}]},{"osd":5,"last update":"Wed Sep 11 08:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.56599999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.66600000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58299999999999996}]},{"osd":6,"last update":"Wed Sep 11 08:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.71699999999999997}]},{"osd":7,"last update":"Wed Sep 11 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56100000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297948,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447204,"statfs":{"total":95995035648,"available":95689936896,"internally_reserved":0,"allocated":606208,"data_stored":234429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.34499999999999997,"15min":0.33600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.376},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.373,"15min":0.36699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.314}]},{"osd":1,"last update":"Wed Sep 11 08:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.39500000000000002},"min":{"1min":0.24099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.35499999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:25:35 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.435,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.51200000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40000000000000002,"15min":0.40400000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.40300000000000002}]},{"osd":4,"last update":"Wed Sep 11 08:24:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68600000000000005}]},{"osd":5,"last update":"Wed Sep 11 08:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.79900000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.625},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.626,"15min":0.58799999999999997},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.9329999999999998,"5min":5.9329999999999998,"15min":5.9329999999999998},"last":0.57299999999999995}]},{"osd":6,"last update":"Wed Sep 11 08:24:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67000000000000004}]},{"osd":7,"last update":"Wed Sep 11 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":5.976,"5min":5.976,"15min":5.976},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.56899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":236214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 11 08:24:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.32000000000000001,"15min":0.308},"min":{"1min":0.191,"5min":0.19,"15min":0.19},"max":{"1min":0.47799999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.35399999999999998,"15min":0.34499999999999997},"min":{"1min":0.219,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.39200000000000002}]},{"osd":2,"last update":"Wed Sep 11 08:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.371,"15min":0.372},"min":{"1min":0.23999999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.504,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.215,"15min":0.215},"max":{"1min":0.56100000000000005,"5min":0.621,"15min":0.621},"last":0.41499999999999998}]},{"osd":3,"last update":"Wed Sep 11 08:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.442,"15min":0.46000000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.624,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.40300000000000002,"15min":0.41699999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36899999999999999}]},{"osd":4,"last update":"Wed Sep 11 08:24:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.54700000000000004,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.70499999999999996}]},{"osd":5,"last update":"Wed Sep 11 08:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56799999999999995,"15min":0.56399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Wed Sep 11 08:24:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.751}]},{"osd":7,"last update":"Wed Sep 11 08:25:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298540,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446612,"statfs":{"total":95995035648,"available":95689330688,"internally_reserved":0,"allocated":1216512,"data_stored":824653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 11 08:24:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.309},"min":{"1min":0.20599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.498,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.34399999999999997,"15min":0.32400000000000001},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.52000000000000002,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.314}]},{"osd":2,"last update":"Wed Sep 11 08:25:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36699999999999999,"15min":0.36499999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.44500000000000001,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.35399999999999998,"15min":0.34100000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.46899999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.24299999999999999}]},{"osd":3,"last update":"Wed Sep 11 08:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.41999999999999998,"15min":0.43099999999999999},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.48799999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.39400000000000002,"15min":0.41099999999999998},"min":{"1min":0.26600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.44400000000000001,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.32600000000000001}]},{"osd":4,"last update":"Wed Sep 11 08:24:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67400000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53300000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47499999999999998}]},{"osd":5,"last update":"Wed Sep 11 08:25:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.41499999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.44700000000000001}]},{"osd":6,"last update":"Wed Sep 11 08:24:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.40999999999999998}]},{"osd":7,"last update":"Wed Sep 11 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:25:47.868 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-11T08:25:47.868 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T08:25:47.868 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-11T08:25:47.868 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-11T08:25:48.123 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:48.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:48 smithi110 ceph-mon[39404]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:25:48.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:48 smithi110 ceph-mon[39404]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:48.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:48 smithi110 ceph-mon[28998]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:25:48.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:48 smithi110 ceph-mon[28998]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:48.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:48 smithi139 ceph-mon[32601]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-11T08:25:48.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:48 smithi139 ceph-mon[32601]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:48.821 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:48 smithi139 bash[63301]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-11T08:25:48.821 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:48 smithi139 bash[63301]: Writing manifest to image destination 2024-09-11T08:25:49.291 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 podman[63301]: 2024-09-11 08:25:48.941197946 +0000 UTC m=+2.241894622 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-11T08:25:49.292 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 podman[63301]: 2024-09-11 08:25:49.095108953 +0000 UTC m=+2.395805674 container create 65651de3087bcecd082b8fafefdd42405c9a414afc507c39cfa0d4f3d4edf4fc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 podman[63301]: 2024-09-11 08:25:49.29198296 +0000 UTC m=+2.592679645 container init 65651de3087bcecd082b8fafefdd42405c9a414afc507c39cfa0d4f3d4edf4fc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 podman[63301]: 2024-09-11 08:25:49.295403855 +0000 UTC m=+2.596100544 container start 65651de3087bcecd082b8fafefdd42405c9a414afc507c39cfa0d4f3d4edf4fc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.300Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.300Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.301Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.301Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.301Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-11T08:25:49.572 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.301Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.301Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.301Z caller=node_exporter.go:117 level=info collector=arp 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.301Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-11T08:25:49.573 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=edac 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-11T08:25:49.574 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=os 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-11T08:25:49.575 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=stat 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=time 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=uname 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-11T08:25:49.576 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-11T08:25:49.577 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b[63375]: ts=2024-09-11T08:25:49.302Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-11T08:25:49.577 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 bash[63301]: 65651de3087bcecd082b8fafefdd42405c9a414afc507c39cfa0d4f3d4edf4fc 2024-09-11T08:25:49.577 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:25:49 smithi139 systemd[1]: Started Ceph node-exporter.b for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:25:49.936 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-11T08:25:49.937 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T08:25:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1947657390' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1947657390' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-11T08:25:50.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:50 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:25:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1947657390' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-11T08:25:50.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:50 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:51.424 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-11T08:25:51.424 INFO:tasks.cephadm:Setup complete, yielding 2024-09-11T08:25:51.424 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-11T08:25:51.437 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi110.front.sepia.ceph.com 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2850e098-7016-11ef-bcea-c7b262605968 -- bash -c 'set -ex 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> do 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> echo "rotating key for $f" 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> K=$(ceph auth get-key $f) 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> NK="$K" 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> ceph orch daemon rotate-key $f 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> while [ "$K" == "$NK" ]; do 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> sleep 5 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> NK=$(ceph auth get-key $f) 2024-09-11T08:25:51.438 DEBUG:teuthology.orchestra.run.smithi110:> done 2024-09-11T08:25:51.439 DEBUG:teuthology.orchestra.run.smithi110:> done 2024-09-11T08:25:51.439 DEBUG:teuthology.orchestra.run.smithi110:> ' 2024-09-11T08:25:51.710 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/mon.a/config 2024-09-11T08:25:52.801 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:52 smithi139 ceph-mon[32601]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:52.811 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:52 smithi110 ceph-mon[39404]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:52.811 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:52 smithi110 ceph-mon[28998]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:53.969 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.0 2024-09-11T08:25:53.969 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:25:53.969 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.0' 2024-09-11T08:25:53.969 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:25:54.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:25:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:25:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:25:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:25:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:25:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.0 2024-09-11T08:25:54.460 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.0 on host 'smithi110' 2024-09-11T08:25:54.471 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:25:54.471 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:25:54.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:54 smithi139 ceph-mon[32601]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:54.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:54 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/28161404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:25:54.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:54 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:54.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:54 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:54.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[28998]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:54.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/28161404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:25:54.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:54.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:54.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[39404]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:54.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/28161404' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:25:54.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:54.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:54 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:55.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:25:55.881 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.0 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:25:55.882 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:55.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:25:55.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.0 2024-09-11T08:25:55.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:55.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:55.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.0 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-11T08:25:55.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:25:56.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:25:56 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:25:56.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:56 smithi139 ceph-mon[32601]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:56.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:56 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-09-11T08:25:56.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:56 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-11T08:25:56.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:56 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-11T08:25:56.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:56 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-11T08:25:56.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:56 smithi139 ceph-mon[32601]: osdmap e63: 8 total, 8 up, 8 in 2024-09-11T08:25:56.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[28998]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:56.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-09-11T08:25:56.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[28998]: osdmap e63: 8 total, 8 up, 8 in 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[39404]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-11T08:25:56.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:56 smithi110 ceph-mon[39404]: osdmap e63: 8 total, 8 up, 8 in 2024-09-11T08:25:57.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:57 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:57.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:57 smithi139 ceph-mon[32601]: osdmap e64: 8 total, 8 up, 8 in 2024-09-11T08:25:57.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:57 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:57.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:57 smithi110 ceph-mon[28998]: osdmap e64: 8 total, 8 up, 8 in 2024-09-11T08:25:57.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:57 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:25:57.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:57 smithi110 ceph-mon[39404]: osdmap e64: 8 total, 8 up, 8 in 2024-09-11T08:25:58.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:58 smithi139 ceph-mon[32601]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:58.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:25:58 smithi139 ceph-mon[32601]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-11T08:25:58.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:58 smithi110 ceph-mon[28998]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:58.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:25:58 smithi110 ceph-mon[28998]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-11T08:25:58.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:58 smithi110 ceph-mon[39404]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:25:58.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:25:58 smithi110 ceph-mon[39404]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-09-11T08:25:59.473 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:25:59.788 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:25:59.789 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:25:59.789 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:00 smithi139 ceph-mon[32601]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:26:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:00 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2356979843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:00 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:00.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[28998]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:26:00.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2356979843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:00.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:00.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:00.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[39404]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-11T08:26:00.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2356979843' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:00.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:00.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:00 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:01.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:01 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:01 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:01 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:01 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-11T08:26:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:01 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-11T08:26:01.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:01 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2024-09-11T08:26:01.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:01.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:01.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:01.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-11T08:26:01.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-11T08:26:01.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2024-09-11T08:26:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-11T08:26:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]: dispatch 2024-09-11T08:26:01.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:01 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.0", "format": "json"}]': finished 2024-09-11T08:26:02.817 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: Rotating authentication key for osd.0 2024-09-11T08:26:02.817 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.0 on smithi110 2024-09-11T08:26:02.817 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-09-11T08:26:02.817 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-09-11T08:26:02.817 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: Cluster is now healthy 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: Rotating authentication key for osd.0 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.0 on smithi110 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: Cluster is now healthy 2024-09-11T08:26:02.818 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.819 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.819 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.819 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:02 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: Rotating authentication key for osd.0 2024-09-11T08:26:02.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.0 on smithi110 2024-09-11T08:26:02.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-09-11T08:26:02.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-09-11T08:26:02.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: Cluster is now healthy 2024-09-11T08:26:02.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:02.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:02 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:03.816 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:03 smithi139 ceph-mon[32601]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-11T08:26:03.816 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:03 smithi139 ceph-mon[32601]: Reconfiguring daemon alertmanager.a on smithi110 2024-09-11T08:26:03.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:03 smithi110 ceph-mon[28998]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-11T08:26:03.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:03 smithi110 ceph-mon[28998]: Reconfiguring daemon alertmanager.a on smithi110 2024-09-11T08:26:03.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:03 smithi110 ceph-mon[39404]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-11T08:26:03.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:03 smithi110 ceph-mon[39404]: Reconfiguring daemon alertmanager.a on smithi110 2024-09-11T08:26:04.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:26:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:04.523 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:04 smithi110 ceph-mon[28998]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-09-11T08:26:04.523 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:04 smithi110 ceph-mon[39404]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-09-11T08:26:04.791 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:04.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:04 smithi139 ceph-mon[32601]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 2 objects/s recovering 2024-09-11T08:26:05.106 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:05.106 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:05.106 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:05.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:05 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2538795654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:05.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:05 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2538795654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:05.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:05 smithi110 systemd[1]: Stopping Ceph alertmanager.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:26:05.824 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:05 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2538795654' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:06.153 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[71483]: ts=2024-09-11T08:26:05.858Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:26:06.153 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:05 smithi110 podman[74108]: 2024-09-11 08:26:05.899350087 +0000 UTC m=+0.166961940 container died 809843c00c611b277ef6da5e863fd92fc97f676d9cea19d6836052d40ab3e719 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:06.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:26:06 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:26:06.500 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 podman[74108]: 2024-09-11 08:26:06.267258019 +0000 UTC m=+0.534869871 container cleanup 809843c00c611b277ef6da5e863fd92fc97f676d9cea19d6836052d40ab3e719 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:06.501 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 bash[74108]: ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a 2024-09-11T08:26:06.754 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:06 smithi110 ceph-mon[28998]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 47 B/s, 1 objects/s recovering 2024-09-11T08:26:06.754 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:06 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:06.754 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:06 smithi110 ceph-mon[39404]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 47 B/s, 1 objects/s recovering 2024-09-11T08:26:06.754 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:06 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:06.754 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 podman[74119]: 2024-09-11 08:26:06.535657059 +0000 UTC m=+0.639651825 container remove 809843c00c611b277ef6da5e863fd92fc97f676d9cea19d6836052d40ab3e719 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:06.755 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 podman[74119]: 2024-09-11 08:26:06.611234126 +0000 UTC m=+0.715228890 volume remove dc3a99b96f86bd161c5406280b02560c5b66e33a6091ab5f4d5f66458990df47 2024-09-11T08:26:06.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:06 smithi139 ceph-mon[32601]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 47 B/s, 1 objects/s recovering 2024-09-11T08:26:06.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:06 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:07.087 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-11T08:26:07.087 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 systemd[1]: Stopped Ceph alertmanager.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:26:07.088 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 systemd[1]: Starting Ceph alertmanager.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:26:07.088 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 podman[74222]: 2024-09-11 08:26:06.87333704 +0000 UTC m=+0.016267660 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-11T08:26:07.088 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:06 smithi110 podman[74222]: 2024-09-11 08:26:06.979031579 +0000 UTC m=+0.121962234 volume create 371cb645370386f5ad8b84407722bafa30e6c6ece930800c7b93b0f32f9a4ffa 2024-09-11T08:26:07.088 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 podman[74222]: 2024-09-11 08:26:07.087793994 +0000 UTC m=+0.230724602 container create a82b469208e5672a099e926d90285602278f5d0a26e4502bcf3c6c5fad2054ba (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:07.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-mon[28998]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 999 B/s rd, 0 op/s; 42 B/s, 1 objects/s recovering 2024-09-11T08:26:07.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-mon[39404]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 999 B/s rd, 0 op/s; 42 B/s, 1 objects/s recovering 2024-09-11T08:26:07.653 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 podman[74222]: 2024-09-11 08:26:07.38294478 +0000 UTC m=+0.525875402 container init a82b469208e5672a099e926d90285602278f5d0a26e4502bcf3c6c5fad2054ba (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:07.653 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 podman[74222]: 2024-09-11 08:26:07.385869601 +0000 UTC m=+0.528800223 container start a82b469208e5672a099e926d90285602278f5d0a26e4502bcf3c6c5fad2054ba (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.398Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.398Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.399Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.110 port=9094 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.400Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.423Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.423Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.426Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:07.426Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 bash[74222]: a82b469208e5672a099e926d90285602278f5d0a26e4502bcf3c6c5fad2054ba 2024-09-11T08:26:07.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:07 smithi110 systemd[1]: Started Ceph alertmanager.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:26:07.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:07 smithi139 ceph-mon[32601]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 999 B/s rd, 0 op/s; 42 B/s, 1 objects/s recovering 2024-09-11T08:26:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:08 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:08 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:08 smithi139 ceph-mon[32601]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-11T08:26:08.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:08 smithi139 ceph-mon[32601]: Reconfiguring daemon prometheus.a on smithi139 2024-09-11T08:26:08.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:08.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:08.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[28998]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-11T08:26:08.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[28998]: Reconfiguring daemon prometheus.a on smithi139 2024-09-11T08:26:08.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:08.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:08.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[39404]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-11T08:26:08.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:08 smithi110 ceph-mon[39404]: Reconfiguring daemon prometheus.a on smithi139 2024-09-11T08:26:09.565 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:09 smithi139 ceph-mon[32601]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-09-11T08:26:09.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:09 smithi110 ceph-mon[28998]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-09-11T08:26:09.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:09 smithi110 ceph-mon[39404]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-09-11T08:26:09.654 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:09 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:09.401Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001087995s 2024-09-11T08:26:10.110 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:10.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:10.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/616715994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:10.822 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 systemd[1]: Stopping Ceph prometheus.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:26:10.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:10.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:10 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/616715994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:10.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:10.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:10 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/616715994' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:11.115 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:26:11.115 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-11T08:26:11.115 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-11T08:26:11.115 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-11T08:26:11.115 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-11T08:26:11.115 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-11T08:26:11.115 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-11T08:26:11.116 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-11T08:26:11.116 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.823Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-11T08:26:11.116 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.824Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-11T08:26:11.116 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.824Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-11T08:26:11.116 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[61135]: ts=2024-09-11T08:26:10.824Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-11T08:26:11.116 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:10 smithi139 podman[64287]: 2024-09-11 08:26:10.833821649 +0000 UTC m=+0.129907879 container died 7edc207477be36ddef2fc969977510017e8813fadfe5563cd80d32e0ea010adc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:11.490 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 podman[64287]: 2024-09-11 08:26:11.222460092 +0000 UTC m=+0.518546309 container cleanup 7edc207477be36ddef2fc969977510017e8813fadfe5563cd80d32e0ea010adc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:11.490 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 bash[64287]: ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a 2024-09-11T08:26:11.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:11 smithi139 ceph-mon[32601]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-09-11T08:26:11.822 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 podman[64302]: 2024-09-11 08:26:11.490207377 +0000 UTC m=+0.658205427 container remove 7edc207477be36ddef2fc969977510017e8813fadfe5563cd80d32e0ea010adc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:11.822 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-11T08:26:11.822 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 systemd[1]: Stopped Ceph prometheus.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:26:11.822 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@prometheus.a.service: Consumed 1.042s CPU time. 2024-09-11T08:26:11.822 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 systemd[1]: Starting Ceph prometheus.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:26:11.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:11 smithi110 ceph-mon[28998]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-09-11T08:26:11.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:11 smithi110 ceph-mon[39404]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-09-11T08:26:12.119 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 podman[64395]: 2024-09-11 08:26:11.757849577 +0000 UTC m=+0.017215809 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-11T08:26:12.120 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:11 smithi139 podman[64395]: 2024-09-11 08:26:11.87480001 +0000 UTC m=+0.134166242 container create c90a41240ab010691965638bf12a4ff52f6c8fa68286ae631017b16d600577bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:12.120 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 podman[64395]: 2024-09-11 08:26:12.093302129 +0000 UTC m=+0.352668361 container init c90a41240ab010691965638bf12a4ff52f6c8fa68286ae631017b16d600577bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:12.120 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 podman[64395]: 2024-09-11 08:26:12.096016238 +0000 UTC m=+0.355382483 container start c90a41240ab010691965638bf12a4ff52f6c8fa68286ae631017b16d600577bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:26:12.572 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.119Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.119Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.119Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi139 (none))" 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.119Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.119Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.121Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.121Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.123Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.123Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.124Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.125Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.458µs 2024-09-11T08:26:12.573 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.125Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.125Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.125Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.125Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.761µs wal_replay_duration=820.615µs wbl_replay_duration=136ns total_replay_duration=869.748µs 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.126Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.126Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.127Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.141Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.921781ms db_storage=808ns remote_storage=1.277µs web_handler=330ns query_engine=602ns scrape=446.088µs scrape_sd=25.299µs notify=23.955µs notify_sd=5.812µs rules=14.148688ms tracing=5.92µs 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.141Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:26:12.141Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 bash[64395]: c90a41240ab010691965638bf12a4ff52f6c8fa68286ae631017b16d600577bd 2024-09-11T08:26:12.574 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:26:12 smithi139 systemd[1]: Started Ceph prometheus.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:26:12.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Bus STOPPING 2024-09-11T08:26:12.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:26:12.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Bus STOPPED 2024-09-11T08:26:12.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Bus STARTING 2024-09-11T08:26:12.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Serving on http://:::9283 2024-09-11T08:26:12.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Bus STARTED 2024-09-11T08:26:12.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Bus STOPPING 2024-09-11T08:26:13.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:26:13.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Bus STOPPED 2024-09-11T08:26:13.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:12 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:12] ENGINE Bus STARTING 2024-09-11T08:26:13.154 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE Serving on http://:::9283 2024-09-11T08:26:13.154 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE Bus STARTED 2024-09-11T08:26:13.154 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE Bus STOPPING 2024-09-11T08:26:13.563 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.563 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.563 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:26:13.563 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:26:13.563 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi110.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi110.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi139.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi139.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi139.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi139.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.564 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi110.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi110.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:26:13.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi139.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi139.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi139.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi139.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.566 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:13.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi110.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi110.front.sepia.ceph.com:9093"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi139.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi139.front.sepia.ceph.com:9095"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-11T08:26:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi139.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:26:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi139.front.sepia.ceph.com:3000"}]: dispatch 2024-09-11T08:26:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:13.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:13 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:13.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:26:13.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE Bus STOPPED 2024-09-11T08:26:13.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE Bus STARTING 2024-09-11T08:26:13.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE Serving on http://:::9283 2024-09-11T08:26:13.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:13 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:26:13] ENGINE Bus STARTED 2024-09-11T08:26:14.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:14 smithi139 ceph-mon[32601]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:14.585 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:14 smithi110 ceph-mon[28998]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:14.585 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:14 smithi110 ceph-mon[39404]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:15.424 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:15.730 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:15.731 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:15.731 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[28998]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:16.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3336142287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[39404]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:16.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:15 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3336142287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:16.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:15 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:16.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:15 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:16.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:15 smithi139 ceph-mon[32601]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:16.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:15 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3336142287' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:16.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:26:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:26:17.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:17.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:17 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:17 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:17 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:17.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:17 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:17.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:17 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:17 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:17.653 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:26:17.403Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002890978s 2024-09-11T08:26:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:17.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:17 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:26:18.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:18 smithi110 ceph-mon[28998]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:18.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:18 smithi110 ceph-mon[39404]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:18.676 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:18 smithi139 ceph-mon[32601]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:20.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:20 smithi139 ceph-mon[32601]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:20.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:20 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:20.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:20 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:20.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:20 smithi110 ceph-mon[28998]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:20.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:20 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:20.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:20 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:20.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:20 smithi110 ceph-mon[39404]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:20.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:20 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:20.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:20 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:20.732 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:21.042 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:21.043 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:21.043 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:21.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:21 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:21 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:21 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:21.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:21 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:21.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:21 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:21 smithi139 ceph-mon[32601]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:21.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:21 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1582521530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:21.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:21.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[28998]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1582521530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[39404]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:21.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:21 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1582521530' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:24.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:26:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:24.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:24 smithi139 ceph-mon[32601]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:24.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:24 smithi110 ceph-mon[28998]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:24.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:24 smithi110 ceph-mon[39404]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:25.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:25 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:25.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:25 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:25.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:25 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:26.044 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:26.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:26 smithi110 ceph-mon[28998]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:26.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:26 smithi110 ceph-mon[39404]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:26.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:25] "GET /metrics HTTP/1.1" 200 219787 "" "Prometheus/2.43.0" 2024-09-11T08:26:26.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:26 smithi139 ceph-mon[32601]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:26.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:26:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:26:26.360 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:26.360 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:26.360 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:27.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:27.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:27 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4253759756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:27.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:27.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:27 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4253759756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:27.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:27.405 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:27 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4253759756' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:28.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:28 smithi139 ceph-mon[32601]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:28.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:28 smithi110 ceph-mon[28998]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:28.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:28 smithi110 ceph-mon[39404]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:30.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:30 smithi139 ceph-mon[32601]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:30.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:30 smithi110 ceph-mon[28998]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:30.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:30 smithi110 ceph-mon[39404]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:31.362 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:31.676 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:32.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:32 smithi139 ceph-mon[32601]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:32.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:32 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4010658787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:32.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:32 smithi110 ceph-mon[28998]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:32.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:32 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4010658787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:32.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:32 smithi110 ceph-mon[39404]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:32.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:32 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4010658787' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:34.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:26:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:34.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:34 smithi139 ceph-mon[32601]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:34.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:34 smithi110 ceph-mon[28998]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:34.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:34 smithi110 ceph-mon[39404]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:36.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:36 smithi110 ceph-mon[28998]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:36.154 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:35] "GET /metrics HTTP/1.1" 200 219787 "" "Prometheus/2.43.0" 2024-09-11T08:26:36.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:36 smithi110 ceph-mon[39404]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:36.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:36 smithi139 ceph-mon[32601]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:36.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:26:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:26:36.679 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:36.994 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:36.994 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:36.994 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:37.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:37 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:37.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:37 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2299150949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:37 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:37 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2299150949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:37.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:37 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:37.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:37 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2299150949' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:38.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:38 smithi110 ceph-mon[28998]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:38.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:38 smithi110 ceph-mon[39404]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:38.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:38 smithi139 ceph-mon[32601]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:40.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:40 smithi110 ceph-mon[28998]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:40.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:40 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:40.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:40 smithi110 ceph-mon[39404]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:40.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:40 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:40.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:40 smithi139 ceph-mon[32601]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:40.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:40 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:41.995 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:42.307 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:42.307 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:42.307 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:42.307 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:42 smithi110 ceph-mon[28998]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:42.308 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:42 smithi110 ceph-mon[39404]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:42.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:42 smithi139 ceph-mon[32601]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:43.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:43 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3755620802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:43.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:43 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3755620802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:43.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:43 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3755620802' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:44.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:26:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:44 smithi110 ceph-mon[28998]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:44.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:44 smithi110 ceph-mon[39404]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:44.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:44 smithi139 ceph-mon[32601]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:46.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:46 smithi110 ceph-mon[28998]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:46.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:46 smithi110 ceph-mon[39404]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:46.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:45] "GET /metrics HTTP/1.1" 200 219756 "" "Prometheus/2.43.0" 2024-09-11T08:26:46.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:46 smithi139 ceph-mon[32601]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:46.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:26:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:26:47.309 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:47.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:47 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:47.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:47 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:47.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:47 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:47.623 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:47.623 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:47.623 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:48.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:48 smithi110 ceph-mon[28998]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:48.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:48 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1192927182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:48.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:48 smithi110 ceph-mon[39404]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:48.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:48 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1192927182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:48.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:48 smithi139 ceph-mon[32601]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:48.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:48 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1192927182' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:50.374 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:50 smithi139 ceph-mon[32601]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:50.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:50 smithi110 ceph-mon[28998]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:50.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:50 smithi110 ceph-mon[39404]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:52 smithi110 ceph-mon[28998]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:52.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:52 smithi110 ceph-mon[39404]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:52.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:52 smithi139 ceph-mon[32601]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:52.625 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:52.938 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:52.938 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:52.939 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:53.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:53 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/272316403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:53.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:53 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/272316403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:53.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:53 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/272316403' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:54.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:26:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:26:54.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:54 smithi110 ceph-mon[28998]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:54.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:54 smithi110 ceph-mon[39404]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:54.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:54 smithi139 ceph-mon[32601]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:55.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:55.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:55.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:26:56.121 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:26:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:26:55] "GET /metrics HTTP/1.1" 200 219754 "" "Prometheus/2.43.0" 2024-09-11T08:26:56.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:56 smithi110 ceph-mon[28998]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:56.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:56 smithi110 ceph-mon[39404]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:56.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:56 smithi139 ceph-mon[32601]: pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:26:56.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:26:56 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:26:57.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:57 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:57.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:57 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:57 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:26:57.940 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:26:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:26:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:26:58.254 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:26:58.254 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:58 smithi110 ceph-mon[28998]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:58.255 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:58 smithi110 ceph-mon[39404]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:58.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:58 smithi139 ceph-mon[32601]: pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:26:59.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:26:59 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1022477078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:59.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:26:59 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1022477078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:26:59.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:26:59 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1022477078' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:00 smithi110 ceph-mon[28998]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:00.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:00 smithi110 ceph-mon[39404]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:00.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:00 smithi139 ceph-mon[32601]: pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:02.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:02 smithi110 ceph-mon[28998]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:02.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:02 smithi110 ceph-mon[39404]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:02.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:02 smithi139 ceph-mon[32601]: pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:03.257 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:03.565 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:03.566 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:03.566 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:04.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:27:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:04 smithi110 ceph-mon[28998]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:04 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3809971913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:04.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:04 smithi110 ceph-mon[39404]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:04.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:04 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3809971913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:04.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:04 smithi139 ceph-mon[32601]: pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:04.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:04 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3809971913' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:05.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:27:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:05] "GET /metrics HTTP/1.1" 200 219754 "" "Prometheus/2.43.0" 2024-09-11T08:27:06.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:06 smithi110 ceph-mon[28998]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:06.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:06 smithi110 ceph-mon[39404]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:06.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:27:06 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:27:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:06 smithi139 ceph-mon[32601]: pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:07.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:07 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:07.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:07 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:07.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:07 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:08.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:08 smithi110 ceph-mon[28998]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:08.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:08 smithi110 ceph-mon[39404]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:08.568 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:08.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:08 smithi139 ceph-mon[32601]: pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:08.883 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:08.883 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:08.883 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:09.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:09 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2526543523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:09.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:09 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2526543523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:09.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:09 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2526543523' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:10.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:10 smithi139 ceph-mon[32601]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:10.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:10.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:10 smithi110 ceph-mon[28998]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:10.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:10.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:10 smithi110 ceph-mon[39404]: pgmap v72: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:10.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:12.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:12 smithi139 ceph-mon[32601]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:12 smithi110 ceph-mon[28998]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:12.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:12 smithi110 ceph-mon[39404]: pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:13.885 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:14.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:27:13 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:14.202 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:14.202 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:14.203 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:14.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:14 smithi139 ceph-mon[32601]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:14.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:14 smithi110 ceph-mon[28998]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:14.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:14 smithi110 ceph-mon[39404]: pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:15.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:15 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3452087053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:15.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:15 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3452087053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:15.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:15 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3452087053' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:16.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:27:15 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:15] "GET /metrics HTTP/1.1" 200 219756 "" "Prometheus/2.43.0" 2024-09-11T08:27:16.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:16 smithi139 ceph-mon[32601]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:16.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:27:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:27:16.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:16 smithi110 ceph-mon[28998]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:16.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:16 smithi110 ceph-mon[39404]: pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:17.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:17.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:18.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:18 smithi139 ceph-mon[32601]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:18.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:18 smithi110 ceph-mon[28998]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:18.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:18 smithi110 ceph-mon[39404]: pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:19.204 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:19.516 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:19.516 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:19.516 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:20.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:20 smithi139 ceph-mon[32601]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:20.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:20 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1227476439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:20.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:20 smithi110 ceph-mon[28998]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:20.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:20 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1227476439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:20 smithi110 ceph-mon[39404]: pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:20 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1227476439' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:21.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:21 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:27:21.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:21 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:27:21.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:21 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:27:21.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:21 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:27:21.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:27:21.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:27:21.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:27:21.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:27:21.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:27:21.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:27:21.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:27:21.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:21 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:27:22.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:22 smithi139 ceph-mon[32601]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:22.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:22 smithi110 ceph-mon[39404]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:22.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:22 smithi110 ceph-mon[28998]: pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:27:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:24.518 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:24.518 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:24 smithi110 ceph-mon[28998]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:24.519 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:24 smithi110 ceph-mon[39404]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:24.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:24 smithi139 ceph-mon[32601]: pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:24.830 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:24.831 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:24.831 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:25.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:25 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3741672945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:25.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:25 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:25.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:25 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3741672945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:25.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:25 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:25 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3741672945' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:25 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:26.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:27:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:25] "GET /metrics HTTP/1.1" 200 219751 "" "Prometheus/2.43.0" 2024-09-11T08:27:26.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:27:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:27:26.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:26 smithi139 ceph-mon[32601]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:26.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:26 smithi110 ceph-mon[28998]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:26.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:26 smithi110 ceph-mon[39404]: pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:27.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:27.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:27.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:28.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:28 smithi139 ceph-mon[32601]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:28.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:28 smithi110 ceph-mon[28998]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:28.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:28 smithi110 ceph-mon[39404]: pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:29.832 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:30.147 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:30.147 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:30.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:30 smithi110 ceph-mon[28998]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:30.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:30 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2710547349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:30.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:30 smithi110 ceph-mon[39404]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:30.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:30 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2710547349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:30.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:30 smithi139 ceph-mon[32601]: pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:30.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:30 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2710547349' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:32.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:32 smithi139 ceph-mon[32601]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:32.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:32 smithi110 ceph-mon[28998]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:32.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:32 smithi110 ceph-mon[39404]: pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:34.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:27:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:34.456 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:34 smithi110 ceph-mon[28998]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:34.456 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:34 smithi110 ceph-mon[39404]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:34.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:34 smithi139 ceph-mon[32601]: pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:35.149 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:35.462 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:35.463 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:35.463 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:36.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:27:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:35] "GET /metrics HTTP/1.1" 200 219751 "" "Prometheus/2.43.0" 2024-09-11T08:27:36.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:27:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:27:36.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:36 smithi139 ceph-mon[32601]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:36.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:36 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1178886117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:36.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:36 smithi110 ceph-mon[28998]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:36.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:36 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1178886117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:36.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:36 smithi110 ceph-mon[39404]: pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:36.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:36 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1178886117' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:37.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:37 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:37.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:37 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:37.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:37 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:38.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:38 smithi139 ceph-mon[32601]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:38.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:38 smithi110 ceph-mon[28998]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:38 smithi110 ceph-mon[39404]: pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:40.464 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:40.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:40 smithi139 ceph-mon[32601]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:40.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:40 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:40.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:40 smithi110 ceph-mon[28998]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:40.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:40 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:40.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:40 smithi110 ceph-mon[39404]: pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:40.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:40 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:41.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:41 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3664638090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:41.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:41 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3664638090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:41.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:41 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3664638090' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:42.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:42 smithi139 ceph-mon[32601]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:42.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:42 smithi110 ceph-mon[28998]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:42.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:42 smithi110 ceph-mon[39404]: pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:44.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:27:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:44.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:44 smithi139 ceph-mon[32601]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:44.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:44 smithi110 ceph-mon[28998]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:44.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:44 smithi110 ceph-mon[39404]: pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:45.779 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:46.094 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:46.094 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:46.094 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:46.094 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:27:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:45] "GET /metrics HTTP/1.1" 200 219761 "" "Prometheus/2.43.0" 2024-09-11T08:27:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:46 smithi110 ceph-mon[28998]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:46 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2413641129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:46.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:46 smithi110 ceph-mon[39404]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:46.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:46 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2413641129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:46.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:46 smithi139 ceph-mon[32601]: pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:46.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:46 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2413641129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:46.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:27:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:27:47.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:47 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:47.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:47 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:47.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:47 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:48.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:48 smithi139 ceph-mon[32601]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:48.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:48 smithi110 ceph-mon[28998]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:48.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:48 smithi110 ceph-mon[39404]: pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:50.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:50 smithi139 ceph-mon[32601]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:50.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:50 smithi110 ceph-mon[28998]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:50 smithi110 ceph-mon[39404]: pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:51.096 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:51.411 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:51.411 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:51.411 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:52.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:52 smithi139 ceph-mon[32601]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:52.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:52 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2115559320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:52.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:52 smithi110 ceph-mon[28998]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:52.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:52 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2115559320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:52.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:52 smithi110 ceph-mon[39404]: pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:52.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:52 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2115559320' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:54.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:27:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:27:54.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:54 smithi139 ceph-mon[32601]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:54.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:54 smithi110 ceph-mon[28998]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:54.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:54 smithi110 ceph-mon[39404]: pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:55.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:55.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:27:56.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:27:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:27:55] "GET /metrics HTTP/1.1" 200 219756 "" "Prometheus/2.43.0" 2024-09-11T08:27:56.413 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:27:56.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:56 smithi139 ceph-mon[32601]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:56.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:27:56 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:27:56.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:56 smithi110 ceph-mon[28998]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:56.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:56 smithi110 ceph-mon[39404]: pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:27:56.729 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:27:56.729 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:27:56.729 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:27:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:57 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:57.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:57 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2904723018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:57 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:57 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2904723018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:57.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:57 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:27:57.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:57 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2904723018' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:27:58.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:27:58 smithi139 ceph-mon[32601]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:58.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:27:58 smithi110 ceph-mon[28998]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:27:58.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:27:58 smithi110 ceph-mon[39404]: pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:00.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:00 smithi139 ceph-mon[32601]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:00.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:00 smithi110 ceph-mon[28998]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:00.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:00 smithi110 ceph-mon[39404]: pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:01.730 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:02.042 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:02.042 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:02.042 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:02.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:02 smithi110 ceph-mon[28998]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:02.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:02 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3272861413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:02.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:02 smithi110 ceph-mon[39404]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:02.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:02 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3272861413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:02.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:02 smithi139 ceph-mon[32601]: pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:02.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:02 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3272861413' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:04.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:28:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:04.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:04 smithi139 ceph-mon[32601]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:04.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:04 smithi110 ceph-mon[28998]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:04.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:04 smithi110 ceph-mon[39404]: pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:06.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:28:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:05] "GET /metrics HTTP/1.1" 200 219756 "" "Prometheus/2.43.0" 2024-09-11T08:28:06.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:06 smithi139 ceph-mon[32601]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:06.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:28:06 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:28:06.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:06 smithi110 ceph-mon[28998]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:06.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:06 smithi110 ceph-mon[39404]: pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:07.044 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:07.357 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:07.358 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:07.358 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:07 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:07.358 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:07.358 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:07 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:07.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:07 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:08.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:08 smithi139 ceph-mon[32601]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:08.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:08 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2898246081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:08.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:08 smithi110 ceph-mon[28998]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:08.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:08 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2898246081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:08.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:08 smithi110 ceph-mon[39404]: pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:08.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:08 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2898246081' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:10.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:10 smithi139 ceph-mon[32601]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:10.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:10.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:10 smithi110 ceph-mon[28998]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:10.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:10.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:10 smithi110 ceph-mon[39404]: pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:10.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:12.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:12 smithi139 ceph-mon[32601]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:12 smithi110 ceph-mon[28998]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:12.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:12 smithi110 ceph-mon[39404]: pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:12.673 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:13.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:13 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3099159641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:13.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:13 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3099159641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:13.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:13 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3099159641' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:14.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:28:13 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:14.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:14 smithi139 ceph-mon[32601]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:14.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:14 smithi110 ceph-mon[28998]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:14.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:14 smithi110 ceph-mon[39404]: pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:16.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:28:15 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:15] "GET /metrics HTTP/1.1" 200 219720 "" "Prometheus/2.43.0" 2024-09-11T08:28:16.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:16 smithi139 ceph-mon[32601]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:16.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:28:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:28:16.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:16 smithi110 ceph-mon[28998]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:16.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:16 smithi110 ceph-mon[39404]: pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:17.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:17.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:17.674 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:18.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:18 smithi139 ceph-mon[32601]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:18.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:18 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3672663016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:18.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:18 smithi110 ceph-mon[39404]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:18.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:18 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3672663016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:18.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:18 smithi110 ceph-mon[28998]: pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:18.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:18 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3672663016' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:20.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:20 smithi139 ceph-mon[32601]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:20 smithi110 ceph-mon[39404]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:20.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:20 smithi110 ceph-mon[28998]: pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:21.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:21 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:28:21.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:21 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:28:21.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:21 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:28:22.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:22 smithi139 ceph-mon[32601]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:28:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:28:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:28:22.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[28998]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:22.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:28:22.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:28:22.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:28:22.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[39404]: pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:22.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:28:22.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:28:22.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:28:22.988 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:23.300 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:23.301 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:23.301 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:23.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:23 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/64729791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:23.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:23 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/64729791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:23.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:23 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/64729791' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:28:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:24.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:24 smithi139 ceph-mon[32601]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:24.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:24 smithi110 ceph-mon[28998]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:24.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:24 smithi110 ceph-mon[39404]: pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:25.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:25 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:25 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:25.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:25 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:26.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:28:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:25] "GET /metrics HTTP/1.1" 200 219722 "" "Prometheus/2.43.0" 2024-09-11T08:28:26.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:26 smithi139 ceph-mon[32601]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:26.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:28:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:28:26.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:26 smithi110 ceph-mon[28998]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:26.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:26 smithi110 ceph-mon[39404]: pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:27.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:27.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:27.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:28.302 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:28.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:28 smithi139 ceph-mon[32601]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:28.615 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:28.615 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:28 smithi110 ceph-mon[39404]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:28.616 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:28 smithi110 ceph-mon[28998]: pgmap v111: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:29.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:29 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4104307217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:29.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:29 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4104307217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:29.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:29 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4104307217' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:30.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:30 smithi110 ceph-mon[28998]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:30.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:30 smithi110 ceph-mon[39404]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:30 smithi139 ceph-mon[32601]: pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:32.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:32 smithi110 ceph-mon[28998]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:32.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:32 smithi110 ceph-mon[39404]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:32.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:32 smithi139 ceph-mon[32601]: pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:33.617 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:33.930 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:34.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:28:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:34.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:34 smithi110 ceph-mon[28998]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:34.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:34 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/868347749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:34.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:34 smithi110 ceph-mon[39404]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:34.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:34 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/868347749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:34 smithi139 ceph-mon[32601]: pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:34.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:34 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/868347749' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:36.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:28:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:35] "GET /metrics HTTP/1.1" 200 219722 "" "Prometheus/2.43.0" 2024-09-11T08:28:36.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:36 smithi139 ceph-mon[32601]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:36.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:28:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:28:36.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:36 smithi110 ceph-mon[28998]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:36.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:36 smithi110 ceph-mon[39404]: pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:37.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:37 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:37.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:37 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:37.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:37 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:38.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:38 smithi110 ceph-mon[28998]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:38 smithi110 ceph-mon[39404]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:38.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:38 smithi139 ceph-mon[32601]: pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:38.931 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:39.246 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:39.246 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:39.246 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:39.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:39 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2870993060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:39.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:39 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2870993060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:39.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:39 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2870993060' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:40.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:40 smithi110 ceph-mon[28998]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:40.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:40 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:40.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:40 smithi110 ceph-mon[39404]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:40.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:40 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:40 smithi139 ceph-mon[32601]: pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:40.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:40 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:42.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:42 smithi110 ceph-mon[28998]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:42.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:42 smithi110 ceph-mon[39404]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:42.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:42 smithi139 ceph-mon[32601]: pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:44.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:28:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:44.248 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:44.562 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:44.563 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:44.563 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:44.563 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:44 smithi110 ceph-mon[28998]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:44.563 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:44 smithi110 ceph-mon[39404]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:44.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:44 smithi139 ceph-mon[32601]: pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:45.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:45 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3288194614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:45.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:45 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3288194614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:45.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:45 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3288194614' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:46.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:28:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:45] "GET /metrics HTTP/1.1" 200 219721 "" "Prometheus/2.43.0" 2024-09-11T08:28:46.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:46 smithi139 ceph-mon[32601]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:46.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:28:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:28:46.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:46 smithi110 ceph-mon[28998]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:46.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:46 smithi110 ceph-mon[39404]: pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:47.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:47 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:47.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:47 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:47.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:47 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:48.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:48 smithi110 ceph-mon[28998]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:48.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:48 smithi110 ceph-mon[39404]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:48.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:48 smithi139 ceph-mon[32601]: pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:49.564 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:49.876 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:49.876 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:49.876 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:50.623 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:50 smithi139 ceph-mon[32601]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:50.623 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:50 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/520352323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:50.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:50 smithi110 ceph-mon[28998]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:50.656 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:50 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/520352323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:50 smithi110 ceph-mon[39404]: pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:50 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/520352323' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:52.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:52 smithi110 ceph-mon[28998]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:52.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:52 smithi110 ceph-mon[39404]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:52.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:52 smithi139 ceph-mon[32601]: pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:54.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:28:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:28:54.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:54 smithi110 ceph-mon[28998]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:54.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:54 smithi110 ceph-mon[39404]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:54.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:54 smithi139 ceph-mon[32601]: pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:54.878 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:28:55.196 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:28:55.197 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:28:55.197 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:28:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:55 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/901039224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:55.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:55.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:55 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/901039224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:55.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:28:55.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:55 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/901039224' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:28:56.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:28:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:28:55] "GET /metrics HTTP/1.1" 200 219726 "" "Prometheus/2.43.0" 2024-09-11T08:28:56.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:56 smithi139 ceph-mon[32601]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:56.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:28:56 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:28:56.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:56 smithi110 ceph-mon[28998]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:56.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:56 smithi110 ceph-mon[39404]: pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:28:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:57 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:57.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:57 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:57.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:57 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:28:58.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:28:58 smithi110 ceph-mon[28998]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:58.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:28:58 smithi110 ceph-mon[39404]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:28:58.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:28:58 smithi139 ceph-mon[32601]: pgmap v126: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:00.198 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:00.513 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:29:00.513 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:29:00.513 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:00.514 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:00 smithi110 ceph-mon[28998]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:00.514 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:00 smithi110 ceph-mon[39404]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:00.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:00 smithi139 ceph-mon[32601]: pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:01.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:01 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2982341942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:01.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:01 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2982341942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:01.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:01 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2982341942' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:02.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:02 smithi110 ceph-mon[28998]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:02.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:02 smithi110 ceph-mon[39404]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:02.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:02 smithi139 ceph-mon[32601]: pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:04.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:29:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:04.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:04 smithi110 ceph-mon[28998]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:04.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:04 smithi110 ceph-mon[39404]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:04.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:04 smithi139 ceph-mon[32601]: pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:05.515 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:29:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:29:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:06.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:29:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:05] "GET /metrics HTTP/1.1" 200 219726 "" "Prometheus/2.43.0" 2024-09-11T08:29:06.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:29:06 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:29:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:06 smithi139 ceph-mon[32601]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:06 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1829521242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:06.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:06 smithi110 ceph-mon[28998]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:06.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:06 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1829521242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:06.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:06 smithi110 ceph-mon[39404]: pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:06.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:06 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1829521242' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:07.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:07 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:07.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:07 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:07.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:07 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:08 smithi139 ceph-mon[32601]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:08.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:08 smithi110 ceph-mon[28998]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:08.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:08 smithi110 ceph-mon[39404]: pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:10.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:10 smithi139 ceph-mon[32601]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:10.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:10.828 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:10 smithi110 ceph-mon[28998]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:10.828 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:10.828 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:10 smithi110 ceph-mon[39404]: pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:10.829 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:29:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:29:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:11.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:11 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2563487167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:11.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:11 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2563487167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:11.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:11 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2563487167' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:12.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:12 smithi139 ceph-mon[32601]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:12.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:12 smithi110 ceph-mon[28998]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:12.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:12 smithi110 ceph-mon[39404]: pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:14.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:29:13 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:14.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:14 smithi139 ceph-mon[32601]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:14.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:14 smithi110 ceph-mon[28998]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:14.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:14 smithi110 ceph-mon[39404]: pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:16.152 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:29:15 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:15] "GET /metrics HTTP/1.1" 200 219729 "" "Prometheus/2.43.0" 2024-09-11T08:29:16.464 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:29:16.464 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:29:16.465 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:16.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:16 smithi139 ceph-mon[32601]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:29:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:29:16.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:16 smithi110 ceph-mon[28998]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:16.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:16 smithi110 ceph-mon[39404]: pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:17.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:17.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:17 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1364576144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:17.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:17.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:17 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1364576144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:17.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:17.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:17 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1364576144' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:18.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:18 smithi139 ceph-mon[32601]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:18.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:18 smithi110 ceph-mon[28998]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:18.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:18 smithi110 ceph-mon[39404]: pgmap v136: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:20.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:20 smithi139 ceph-mon[32601]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:20.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:20 smithi110 ceph-mon[28998]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:20.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:20 smithi110 ceph-mon[39404]: pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:21.466 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:21.788 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:29:21.788 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:29:21.788 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:22 smithi139 ceph-mon[32601]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:22 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2297782401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:22.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[28998]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:22.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:22.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2297782401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:22.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:22.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:22.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:22.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[39404]: pgmap v138: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:22.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:22.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2297782401' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:22.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:22.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:22.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:29:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:24.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:24 smithi139 ceph-mon[32601]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:24.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:24 smithi110 ceph-mon[28998]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:24.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:24 smithi110 ceph-mon[39404]: pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:25.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:25 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:25.861 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:25 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:25.861 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:25 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:26.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:29:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:25] "GET /metrics HTTP/1.1" 200 219727 "" "Prometheus/2.43.0" 2024-09-11T08:29:26.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:26 smithi139 ceph-mon[32601]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:26.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:29:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:29:26.790 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:26.791 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:26 smithi110 ceph-mon[28998]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:26.791 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:26 smithi110 ceph-mon[39404]: pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:27.106 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:29:27.106 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:29:27.106 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:27.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:27.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:27 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3149395440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:27.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:27.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:27 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3149395440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:27.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:27.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:27 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3149395440' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:28.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:28 smithi139 ceph-mon[32601]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:28.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:28 smithi110 ceph-mon[28998]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:28 smithi110 ceph-mon[39404]: pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:30.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:30 smithi139 ceph-mon[32601]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:30 smithi110 ceph-mon[28998]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:30.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:30 smithi110 ceph-mon[39404]: pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:32.108 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== 2024-09-11T08:29:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== ']' 2024-09-11T08:29:32.419 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:32.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:32 smithi139 ceph-mon[32601]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:32.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:32 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3083022518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:32.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:32 smithi110 ceph-mon[28998]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:32.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:32 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3083022518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:32.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:32 smithi110 ceph-mon[39404]: pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:32.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:32 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3083022518' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:34.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:29:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:34 smithi139 ceph-mon[32601]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:34.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:34 smithi110 ceph-mon[28998]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:34 smithi110 ceph-mon[39404]: pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:36.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:29:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:35] "GET /metrics HTTP/1.1" 200 219727 "" "Prometheus/2.43.0" 2024-09-11T08:29:36.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:36 smithi139 ceph-mon[32601]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:36.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:29:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:29:36.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:36 smithi110 ceph-mon[28998]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:36.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:36 smithi110 ceph-mon[39404]: pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:37.421 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.0 2024-09-11T08:29:37.737 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.1 2024-09-11T08:29:37.737 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAYVOFm2AitJBAAiTYgQuAmmElFXnToYzirZw== 2024-09-11T08:29:37.737 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDxUuFm9TGyJxAAgHtBM5FHcylDSFXjat/Z9g== == AQAYVOFm2AitJBAAiTYgQuAmmElFXnToYzirZw== ']' 2024-09-11T08:29:37.738 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:29:37.738 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.1' 2024-09-11T08:29:37.738 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:37 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:37.738 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:37 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:37.738 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.1 2024-09-11T08:29:37.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:37 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:38.037 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== 2024-09-11T08:29:38.037 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== 2024-09-11T08:29:38.037 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.1 2024-09-11T08:29:38.291 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.1 on host 'smithi110' 2024-09-11T08:29:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== == AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== ']' 2024-09-11T08:29:38.303 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3416149746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4182282278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3416149746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4182282278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:38.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:38.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:38.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-11T08:29:38.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-11T08:29:38.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:38 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2024-09-11T08:29:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: pgmap v146: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3416149746' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.0"}]: dispatch 2024-09-11T08:29:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4182282278' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:29:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:29:38.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:29:38.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:38.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-11T08:29:38.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]: dispatch 2024-09-11T08:29:38.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:38 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.1", "format": "json"}]': finished 2024-09-11T08:29:39.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[39404]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:29:39.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.1 2024-09-11T08:29:39.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[39404]: Rotating authentication key for osd.1 2024-09-11T08:29:39.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.1 on smithi110 2024-09-11T08:29:39.776 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[28998]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:29:39.777 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.1 2024-09-11T08:29:39.777 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[28998]: Rotating authentication key for osd.1 2024-09-11T08:29:39.777 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:39 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.1 on smithi110 2024-09-11T08:29:39.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:39 smithi139 ceph-mon[32601]: from='client.24914 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:29:39.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:39 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.1 2024-09-11T08:29:39.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:39 smithi139 ceph-mon[32601]: Rotating authentication key for osd.1 2024-09-11T08:29:39.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:39 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.1 on smithi110 2024-09-11T08:29:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:40 smithi139 ceph-mon[32601]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:40 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:40.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:40 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[28998]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[39404]: pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:40.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:40.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:40.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:40 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:29:42.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:42 smithi139 ceph-mon[32601]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:42.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:42 smithi110 ceph-mon[28998]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:42.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:42 smithi110 ceph-mon[39404]: pgmap v148: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:43.305 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.1 2024-09-11T08:29:43.617 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== 2024-09-11T08:29:43.617 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== == AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== ']' 2024-09-11T08:29:43.617 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:44.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:29:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:44.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:44 smithi139 ceph-mon[32601]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:44.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:44 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4138516115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:44.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:44 smithi110 ceph-mon[28998]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:44.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:44 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4138516115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:44.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:44 smithi110 ceph-mon[39404]: pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:44.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:44 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4138516115' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:46.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:29:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:45] "GET /metrics HTTP/1.1" 200 219725 "" "Prometheus/2.43.0" 2024-09-11T08:29:46.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:46 smithi139 ceph-mon[32601]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:46.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:29:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:29:46.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:46 smithi110 ceph-mon[28998]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:46.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:46 smithi110 ceph-mon[39404]: pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:47.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:47 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:47.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:47 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:47.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:47 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:48.618 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.1 2024-09-11T08:29:48.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:48 smithi139 ceph-mon[32601]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:48.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:48 smithi110 ceph-mon[28998]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:48.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:48 smithi110 ceph-mon[39404]: pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:48.931 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== 2024-09-11T08:29:48.931 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== == AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== ']' 2024-09-11T08:29:48.931 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:49.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:49 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2899297015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:49.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:49 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2899297015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:49.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:49 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2899297015' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:50.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:50 smithi139 ceph-mon[32601]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:50.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:50 smithi110 ceph-mon[28998]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:50.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:50 smithi110 ceph-mon[39404]: pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:52.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:52 smithi139 ceph-mon[32601]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:52.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:52 smithi110 ceph-mon[28998]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:52.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:52 smithi110 ceph-mon[39404]: pgmap v153: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:53.933 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.1 2024-09-11T08:29:54.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:29:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:29:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== 2024-09-11T08:29:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== == AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== ']' 2024-09-11T08:29:54.247 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:29:54.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:54 smithi139 ceph-mon[32601]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:54.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:54 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/235182458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:54.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:54 smithi110 ceph-mon[39404]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:54.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:54 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/235182458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:54.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:54 smithi110 ceph-mon[28998]: pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:54.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:54 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/235182458' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:29:55.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:55.861 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:55.861 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:29:56.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:29:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:29:55] "GET /metrics HTTP/1.1" 200 219738 "" "Prometheus/2.43.0" 2024-09-11T08:29:56.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:56 smithi139 ceph-mon[32601]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:56.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:29:56 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:29:56.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:56 smithi110 ceph-mon[28998]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:56.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:56 smithi110 ceph-mon[39404]: pgmap v155: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:29:57.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:57 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:57.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:57 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:57.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:57 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:29:58.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:29:58 smithi139 ceph-mon[32601]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:58.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:29:58 smithi110 ceph-mon[28998]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:58.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:29:58 smithi110 ceph-mon[39404]: pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:29:59.248 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.1 2024-09-11T08:29:59.561 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== 2024-09-11T08:29:59.562 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== == AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== ']' 2024-09-11T08:29:59.562 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:00.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:00 smithi139 ceph-mon[32601]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:00.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:00 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2268482095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:30:00.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:00 smithi139 ceph-mon[32601]: overall HEALTH_OK 2024-09-11T08:30:00.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:00 smithi110 ceph-mon[28998]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:00.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:00 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2268482095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:30:00.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:00 smithi110 ceph-mon[28998]: overall HEALTH_OK 2024-09-11T08:30:00.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:00 smithi110 ceph-mon[39404]: pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:00.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:00 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2268482095' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:30:00.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:00 smithi110 ceph-mon[39404]: overall HEALTH_OK 2024-09-11T08:30:02.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:02 smithi139 ceph-mon[32601]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:02.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:02 smithi110 ceph-mon[39404]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:02.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:02 smithi110 ceph-mon[28998]: pgmap v158: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:04.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:30:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:04.563 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.1 2024-09-11T08:30:04.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:04 smithi139 ceph-mon[32601]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:04.886 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:04 smithi110 ceph-mon[39404]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:04.887 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:04 smithi110 ceph-mon[28998]: pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:04.887 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDyVOFmA7AnFhAAzCobGFZm+BGeYMuIwDiIDQ== 2024-09-11T08:30:04.887 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.2 2024-09-11T08:30:04.887 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAIU+FmXsatFBAA6Q+oKrflGMTNLGQsc4lgag== == AQDyVOFmA7AnFhAAzCobGFZm+BGeYMuIwDiIDQ== ']' 2024-09-11T08:30:04.887 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:30:04.888 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.2' 2024-09-11T08:30:04.888 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.2 2024-09-11T08:30:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== 2024-09-11T08:30:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== 2024-09-11T08:30:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.2 2024-09-11T08:30:05.467 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.2 on host 'smithi110' 2024-09-11T08:30:05.480 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== == AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== ']' 2024-09-11T08:30:05.480 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:05.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:05 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2542899371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:30:05.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:05 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/273810042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:05.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:05 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:05.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:05 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:05.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:05 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:05.861 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2542899371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:30:05.861 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/273810042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:05.861 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:05.861 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:05.861 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:05.862 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2542899371' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.1"}]: dispatch 2024-09-11T08:30:05.862 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/273810042' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:05.862 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:05.862 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:05.862 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:06.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:30:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:05] "GET /metrics HTTP/1.1" 200 219738 "" "Prometheus/2.43.0" 2024-09-11T08:30:06.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.2 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-11T08:30:06.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:06 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2024-09-11T08:30:06.835 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:06.835 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:06.835 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.2 2024-09-11T08:30:06.835 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.835 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.835 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:06.835 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: pgmap v160: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.2 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:06.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:06.837 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:06.837 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-11T08:30:06.837 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]: dispatch 2024-09-11T08:30:06.837 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:06 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.2", "format": "json"}]': finished 2024-09-11T08:30:07.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:07 smithi139 ceph-mon[32601]: Rotating authentication key for osd.2 2024-09-11T08:30:07.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:07 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.2 on smithi110 2024-09-11T08:30:07.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:07 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:07.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:07 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:07.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:07 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:07.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[39404]: Rotating authentication key for osd.2 2024-09-11T08:30:07.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.2 on smithi110 2024-09-11T08:30:07.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:07.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:07.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:07.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[28998]: Rotating authentication key for osd.2 2024-09-11T08:30:07.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.2 on smithi110 2024-09-11T08:30:07.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:07.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:07.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:07 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:08 smithi139 ceph-mon[32601]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:08 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:08 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:08.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:08 smithi110 ceph-mon[39404]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:08.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:08 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:08.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:08 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:08.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:08 smithi110 ceph-mon[28998]: pgmap v161: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:08.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:08 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:08.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:08 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:10.483 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.2 2024-09-11T08:30:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== 2024-09-11T08:30:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== == AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== ']' 2024-09-11T08:30:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:10.799 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:10 smithi110 ceph-mon[28998]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:10.799 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:10.799 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:10 smithi110 ceph-mon[39404]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:10.800 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:10.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:10 smithi139 ceph-mon[32601]: pgmap v162: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:10.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:11.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:11 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/487030845' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:11.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:11 smithi139 ceph-mon[32601]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:11.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:11 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/487030845' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:11.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:11 smithi110 ceph-mon[39404]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:11.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:11 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/487030845' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:11.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:11 smithi110 ceph-mon[28998]: pgmap v163: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:14.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:30:13 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:14.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:14 smithi110 ceph-mon[39404]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:14.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:14 smithi110 ceph-mon[28998]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:14.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:14 smithi139 ceph-mon[32601]: pgmap v164: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:15.801 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.2 2024-09-11T08:30:16.062 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:30:15 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:15] "GET /metrics HTTP/1.1" 200 219759 "" "Prometheus/2.43.0" 2024-09-11T08:30:16.122 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== 2024-09-11T08:30:16.123 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== == AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== ']' 2024-09-11T08:30:16.123 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:16.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:16 smithi110 ceph-mon[39404]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:16.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:16 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3712948520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:16.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:16 smithi110 ceph-mon[28998]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:16.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:16 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3712948520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:30:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:30:16.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:16 smithi139 ceph-mon[32601]: pgmap v165: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:16.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:16 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3712948520' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:17.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:17.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:17.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:18.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:18 smithi110 ceph-mon[28998]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:18.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:18 smithi110 ceph-mon[39404]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:18.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:18 smithi139 ceph-mon[32601]: pgmap v166: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:20.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:20 smithi110 ceph-mon[28998]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:20.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:20 smithi110 ceph-mon[39404]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:20.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:20 smithi139 ceph-mon[32601]: pgmap v167: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:21.124 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.2 2024-09-11T08:30:21.441 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.3 2024-09-11T08:30:21.441 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQANVeFm/KW5MhAAy7Nc9KkV8+Ailo6QsjWnzw== 2024-09-11T08:30:21.442 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQAeU+Fm5XwRJhAA3dz/UbKK6DUnkaXvLiJDQA== == AQANVeFm/KW5MhAAy7Nc9KkV8+Ailo6QsjWnzw== ']' 2024-09-11T08:30:21.442 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:30:21.442 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.3' 2024-09-11T08:30:21.442 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.3 2024-09-11T08:30:21.742 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== 2024-09-11T08:30:21.742 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== 2024-09-11T08:30:21.742 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.3 2024-09-11T08:30:21.996 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.3 on host 'smithi110' 2024-09-11T08:30:22.007 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== == AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== ']' 2024-09-11T08:30:22.007 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:22.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[28998]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:22.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2937582154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:22.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1152542165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:22.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:22.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:22.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:22.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[39404]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:22.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2937582154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:22.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1152542165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:22.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:22.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:22.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:22.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:22 smithi139 ceph-mon[32601]: pgmap v168: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:22.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:22 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2937582154' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.2"}]: dispatch 2024-09-11T08:30:22.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:22 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1152542165' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:22.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.3 2024-09-11T08:30:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: Rotating authentication key for osd.3 2024-09-11T08:30:23.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.3 on smithi110 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.3 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:23.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: Rotating authentication key for osd.3 2024-09-11T08:30:23.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-11T08:30:23.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-11T08:30:23.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2024-09-11T08:30:23.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:23 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.3 on smithi110 2024-09-11T08:30:23.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:23.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.3 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: Rotating authentication key for osd.3 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]: dispatch 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.3", "format": "json"}]': finished 2024-09-11T08:30:23.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.3 on smithi110 2024-09-11T08:30:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:30:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:24.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:24 smithi139 ceph-mon[32601]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:24.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:24 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:24 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:24 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:24 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[28998]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[39404]: pgmap v169: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:24.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:24 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:25.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:25 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:25.862 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:25 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:25.863 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:25 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:26.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:30:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:25] "GET /metrics HTTP/1.1" 200 219766 "" "Prometheus/2.43.0" 2024-09-11T08:30:26.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:26 smithi139 ceph-mon[32601]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:26.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:30:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:30:26.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:26 smithi110 ceph-mon[28998]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:26.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:26 smithi110 ceph-mon[39404]: pgmap v170: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:27.011 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.3 2024-09-11T08:30:27.326 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== 2024-09-11T08:30:27.326 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== == AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== ']' 2024-09-11T08:30:27.326 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:27.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:27.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:27 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2467516826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:27.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:27.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:27 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2467516826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:27.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:27.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:27 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2467516826' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:28.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:28 smithi139 ceph-mon[32601]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:28.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:28 smithi110 ceph-mon[28998]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:28 smithi110 ceph-mon[39404]: pgmap v171: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:30.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:30 smithi139 ceph-mon[32601]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:30.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:30 smithi110 ceph-mon[39404]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:30 smithi110 ceph-mon[28998]: pgmap v172: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.3 2024-09-11T08:30:32.643 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== 2024-09-11T08:30:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== == AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== ']' 2024-09-11T08:30:32.644 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:32.644 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:32 smithi110 ceph-mon[39404]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:32.644 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:32 smithi110 ceph-mon[28998]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:32.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:32 smithi139 ceph-mon[32601]: pgmap v173: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:33.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:33 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2865096685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:33.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:33 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2865096685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:33.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:33 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2865096685' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:34.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:30:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:34.804 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:34 smithi110 ceph-mon[28998]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:34.805 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:34 smithi110 ceph-mon[39404]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:34 smithi139 ceph-mon[32601]: pgmap v174: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:36.068 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:30:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:35] "GET /metrics HTTP/1.1" 200 219766 "" "Prometheus/2.43.0" 2024-09-11T08:30:36.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:30:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:30:36.573 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:36 smithi139 ceph-mon[32601]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:36.902 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:36 smithi110 ceph-mon[28998]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:36.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:36 smithi110 ceph-mon[39404]: pgmap v175: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:37.646 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.3 2024-09-11T08:30:37.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:37 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:37.902 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:37 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:37.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:37 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:37.950 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== 2024-09-11T08:30:37.950 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== == AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== ']' 2024-09-11T08:30:37.950 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:38.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:38 smithi139 ceph-mon[32601]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:38.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:38 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2319555706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:38.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:38 smithi110 ceph-mon[28998]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:38.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:38 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2319555706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:38.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:38 smithi110 ceph-mon[39404]: pgmap v176: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:38.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:38 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2319555706' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:40 smithi139 ceph-mon[32601]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:40.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:40 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:40.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:40 smithi110 ceph-mon[39404]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:40.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:40 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:40 smithi110 ceph-mon[28998]: pgmap v177: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:40.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:40 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:42.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:42 smithi139 ceph-mon[32601]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:42.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:42 smithi110 ceph-mon[39404]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:42.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:42 smithi110 ceph-mon[28998]: pgmap v178: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:42.952 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.3 2024-09-11T08:30:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== 2024-09-11T08:30:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== == AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== ']' 2024-09-11T08:30:43.269 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:43.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:43 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1020703798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:43.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:43 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1020703798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:43.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:43 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1020703798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:44.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:30:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:44.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:44 smithi139 ceph-mon[32601]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:44.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:44 smithi110 ceph-mon[28998]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:44.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:44 smithi110 ceph-mon[39404]: pgmap v179: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:46.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:30:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:45] "GET /metrics HTTP/1.1" 200 219763 "" "Prometheus/2.43.0" 2024-09-11T08:30:46.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:46 smithi139 ceph-mon[32601]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:46.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:30:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:30:46.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:46 smithi110 ceph-mon[28998]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:46.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:46 smithi110 ceph-mon[39404]: pgmap v180: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:47.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:47 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:47.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:47 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:47.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:47 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:48.270 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.3 2024-09-11T08:30:48.583 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.4 2024-09-11T08:30:48.583 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQAeVeFmKiA7GxAAPHfIxKw7kdqd1ovaXHk5eg== 2024-09-11T08:30:48.583 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQA1U+Fm/TikDBAAK7yd2P3MrzdvlAMPkiMw4A== == AQAeVeFmKiA7GxAAPHfIxKw7kdqd1ovaXHk5eg== ']' 2024-09-11T08:30:48.583 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:30:48.583 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.4' 2024-09-11T08:30:48.583 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:48 smithi110 ceph-mon[28998]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:48.584 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:48 smithi110 ceph-mon[39404]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:48.584 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.4 2024-09-11T08:30:48.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:48 smithi139 ceph-mon[32601]: pgmap v181: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== 2024-09-11T08:30:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== 2024-09-11T08:30:48.881 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.4 2024-09-11T08:30:49.135 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.4 on host 'smithi139' 2024-09-11T08:30:49.146 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== == AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== ']' 2024-09-11T08:30:49.147 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:49.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3427623326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:49.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1163646001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:30:49.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-11T08:30:49.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:49 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2024-09-11T08:30:49.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3427623326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:49.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1163646001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:30:49.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3427623326' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.3"}]: dispatch 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1163646001' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:30:49.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]: dispatch 2024-09-11T08:30:49.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:49 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.4", "format": "json"}]': finished 2024-09-11T08:30:50.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:50 smithi139 ceph-mon[32601]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:50.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:50 smithi139 ceph-mon[32601]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:50 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.4 2024-09-11T08:30:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:50 smithi139 ceph-mon[32601]: Rotating authentication key for osd.4 2024-09-11T08:30:50.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:50 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.4 on smithi139 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[28998]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[28998]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.4 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[28998]: Rotating authentication key for osd.4 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.4 on smithi139 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[39404]: pgmap v182: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[39404]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:30:50.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.4 2024-09-11T08:30:50.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[39404]: Rotating authentication key for osd.4 2024-09-11T08:30:50.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:50 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.4 on smithi139 2024-09-11T08:30:52.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:52 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:52 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:52 smithi139 ceph-mon[32601]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:52 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.322 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:52 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[28998]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[39404]: pgmap v183: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:52.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:52 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:30:54.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:30:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:30:54.149 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.4 2024-09-11T08:30:54.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:54 smithi110 ceph-mon[28998]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:54.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:54 smithi110 ceph-mon[39404]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:54.463 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== 2024-09-11T08:30:54.463 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== == AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== ']' 2024-09-11T08:30:54.463 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:30:54.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:54 smithi139 ceph-mon[32601]: pgmap v184: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:55.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:55 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2336036505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:30:55.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:55.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:55 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2336036505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:30:55.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:55.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:55 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2336036505' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:30:55.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:30:56.129 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:30:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:30:55] "GET /metrics HTTP/1.1" 200 219772 "" "Prometheus/2.43.0" 2024-09-11T08:30:56.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:56 smithi110 ceph-mon[28998]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:56.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:56 smithi110 ceph-mon[39404]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:56.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:56 smithi139 ceph-mon[32601]: pgmap v185: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:30:56.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:30:56 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:30:57.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:57 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:57.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:57 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:57 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:30:58.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:30:58 smithi110 ceph-mon[28998]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:58.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:30:58 smithi110 ceph-mon[39404]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:58.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:30:58 smithi139 ceph-mon[32601]: pgmap v186: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:30:59.465 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.4 2024-09-11T08:30:59.777 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== 2024-09-11T08:30:59.777 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== == AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== ']' 2024-09-11T08:30:59.777 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:00 smithi110 ceph-mon[28998]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:00.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:00 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3220764751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:00.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:00 smithi110 ceph-mon[39404]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:00.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:00 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3220764751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:00.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:00 smithi139 ceph-mon[32601]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:00.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:00 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3220764751' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:02.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:02 smithi110 ceph-mon[39404]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:02.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:02 smithi110 ceph-mon[28998]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:02.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:02 smithi139 ceph-mon[32601]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:04.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:31:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:04.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:04 smithi110 ceph-mon[28998]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:04.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:04 smithi110 ceph-mon[39404]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:04.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:04 smithi139 ceph-mon[32601]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:04.779 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.4 2024-09-11T08:31:05.094 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== 2024-09-11T08:31:05.094 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== == AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== ']' 2024-09-11T08:31:05.094 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:05.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:05 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1420472796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:05.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:05 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1420472796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:05.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:05 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1420472796' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:06.151 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:31:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:05] "GET /metrics HTTP/1.1" 200 219772 "" "Prometheus/2.43.0" 2024-09-11T08:31:06.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:06 smithi110 ceph-mon[28998]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:06.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:06 smithi110 ceph-mon[39404]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:06.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:06 smithi139 ceph-mon[32601]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:06.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:31:06 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:31:07.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:07 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:07.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:07 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:07.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:07 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:08.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:08 smithi139 ceph-mon[32601]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:08.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:08 smithi110 ceph-mon[28998]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:08.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:08 smithi110 ceph-mon[39404]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:10.096 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.4 2024-09-11T08:31:10.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:10 smithi110 ceph-mon[28998]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:10.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:10.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:10 smithi110 ceph-mon[39404]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:10.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:10.410 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.5 2024-09-11T08:31:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQA5VeFmRkMrHhAAys8ohuMlcOJGa+689H6mcw== 2024-09-11T08:31:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBKU+FmCj5FKxAANyuklLan0HoSJ//ajTC1Ag== == AQA5VeFmRkMrHhAAys8ohuMlcOJGa+689H6mcw== ']' 2024-09-11T08:31:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:31:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.5' 2024-09-11T08:31:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.5 2024-09-11T08:31:10.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:10 smithi139 ceph-mon[32601]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:10.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:10.714 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== 2024-09-11T08:31:10.714 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== 2024-09-11T08:31:10.715 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.5 2024-09-11T08:31:10.970 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.5 on host 'smithi139' 2024-09-11T08:31:10.981 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== == AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== ']' 2024-09-11T08:31:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1427417909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/911741105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-11T08:31:11.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1427417909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/911741105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-11T08:31:11.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-11T08:31:11.405 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:11 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2024-09-11T08:31:11.425 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1427417909' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.4"}]: dispatch 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/911741105' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]: dispatch 2024-09-11T08:31:11.426 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:11 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.5", "format": "json"}]': finished 2024-09-11T08:31:12.460 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:12 smithi139 ceph-mon[32601]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:12.460 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:12 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.5 2024-09-11T08:31:12.460 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:12 smithi139 ceph-mon[32601]: Rotating authentication key for osd.5 2024-09-11T08:31:12.460 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:12 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.5 on smithi139 2024-09-11T08:31:12.461 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:12 smithi139 ceph-mon[32601]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[28998]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.5 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[28998]: Rotating authentication key for osd.5 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.5 on smithi139 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[28998]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[39404]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.5", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.5 2024-09-11T08:31:12.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[39404]: Rotating authentication key for osd.5 2024-09-11T08:31:12.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.5 on smithi139 2024-09-11T08:31:12.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:12 smithi110 ceph-mon[39404]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:13.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[39404]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:13.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:13.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:13 smithi110 ceph-mon[28998]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:14.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:31:13 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:13 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:14.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:13 smithi139 ceph-mon[32601]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:15.983 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.5 2024-09-11T08:31:16.128 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:31:15 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:15] "GET /metrics HTTP/1.1" 200 219768 "" "Prometheus/2.43.0" 2024-09-11T08:31:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== 2024-09-11T08:31:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== == AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== ']' 2024-09-11T08:31:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:16.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:16 smithi110 ceph-mon[28998]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:16.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:16 smithi110 ceph-mon[39404]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:16.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:16 smithi139 ceph-mon[32601]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:31:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:31:17.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:17 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1388017798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:17.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:17 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1388017798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:17.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:17 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1388017798' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:17.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:18.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:18 smithi139 ceph-mon[32601]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:18.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:18 smithi110 ceph-mon[28998]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:18.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:18 smithi110 ceph-mon[39404]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:20.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:20 smithi139 ceph-mon[32601]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:20.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:20 smithi110 ceph-mon[28998]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:20 smithi110 ceph-mon[39404]: pgmap v197: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:21.310 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.5 2024-09-11T08:31:21.622 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== 2024-09-11T08:31:21.622 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== == AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== ']' 2024-09-11T08:31:21.622 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:22.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:22 smithi139 ceph-mon[32601]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:22.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:22 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1560251527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:22.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:22 smithi110 ceph-mon[28998]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:22.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:22 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1560251527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:22.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:22 smithi110 ceph-mon[39404]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:22.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:22 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1560251527' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:24.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:31:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:24.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:24 smithi139 ceph-mon[32601]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:24.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:24 smithi110 ceph-mon[28998]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:24.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:24 smithi110 ceph-mon[39404]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:25.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:25 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:25 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:25.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:25 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:26.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:31:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:25] "GET /metrics HTTP/1.1" 200 219774 "" "Prometheus/2.43.0" 2024-09-11T08:31:26.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:26 smithi139 ceph-mon[32601]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:26.572 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:31:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:31:26.624 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.5 2024-09-11T08:31:26.624 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:26 smithi110 ceph-mon[28998]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:26.624 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:26 smithi110 ceph-mon[39404]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:26.940 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== 2024-09-11T08:31:26.940 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== == AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== ']' 2024-09-11T08:31:26.940 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:27.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:27.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:27 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2288357129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:27.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:27.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:27 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2288357129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:27.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:27.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:27 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2288357129' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:28.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:28 smithi139 ceph-mon[32601]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:28.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:28 smithi110 ceph-mon[28998]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:28.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:28 smithi110 ceph-mon[39404]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:30.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:30 smithi139 ceph-mon[32601]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:30.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:30 smithi110 ceph-mon[28998]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:30.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:30 smithi110 ceph-mon[39404]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:31.942 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.5 2024-09-11T08:31:32.256 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.6 2024-09-11T08:31:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBPVeFm/eMlAxAAj+OgKKePaE3Z9VyGd5MMwQ== 2024-09-11T08:31:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQBiU+FmI3IPHRAAb+0lR0V2nZD6ELQ3CQSPtA== == AQBPVeFm/eMlAxAAj+OgKKePaE3Z9VyGd5MMwQ== ']' 2024-09-11T08:31:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:31:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.6' 2024-09-11T08:31:32.256 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.6 2024-09-11T08:31:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== 2024-09-11T08:31:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== 2024-09-11T08:31:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.6 2024-09-11T08:31:32.557 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:32 smithi110 ceph-mon[28998]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:32.558 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:32 smithi110 ceph-mon[39404]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:32.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:32 smithi139 ceph-mon[32601]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:32.812 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.6 on host 'smithi139' 2024-09-11T08:31:32.823 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== == AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== ']' 2024-09-11T08:31:32.823 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:33.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1099155142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:33.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2809124762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:33.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:33.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:33.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:33.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1099155142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2809124762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1099155142' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.5"}]: dispatch 2024-09-11T08:31:33.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2809124762' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:33.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:33.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:33.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:33.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:33 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:33.871 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:31:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:34.221 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:34.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.6 2024-09-11T08:31:34.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:34.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: Rotating authentication key for osd.6 2024-09-11T08:31:34.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-11T08:31:34.222 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-11T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.6 2024-09-11T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: Rotating authentication key for osd.6 2024-09-11T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-11T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-11T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.6 on smithi139 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.6", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.6 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: Rotating authentication key for osd.6 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]: dispatch 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2024-09-11T08:31:34.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:34 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.6 on smithi139 2024-09-11T08:31:34.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.6", "format": "json"}]': finished 2024-09-11T08:31:34.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:34 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.6 on smithi139 2024-09-11T08:31:36.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:35 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.071 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:35 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:35 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:35 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:35 smithi139 ceph-mon[32601]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:36.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:35] "GET /metrics HTTP/1.1" 200 219774 "" "Prometheus/2.43.0" 2024-09-11T08:31:36.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.153 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[28998]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:36.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:36.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:35 smithi110 ceph-mon[39404]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:36.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:31:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:31:37.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:36 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:37.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:36 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:37.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:36 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:37.825 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.6 2024-09-11T08:31:38.139 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== 2024-09-11T08:31:38.139 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== == AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== ']' 2024-09-11T08:31:38.140 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:38.140 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:37 smithi110 ceph-mon[39404]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:38.140 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:37 smithi110 ceph-mon[28998]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:38.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:37 smithi139 ceph-mon[32601]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:39.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:38 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3726488723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:39.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:38 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3726488723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:39.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:38 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3726488723' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:40.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:39 smithi139 ceph-mon[32601]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:40.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:39 smithi110 ceph-mon[39404]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:40.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:39 smithi110 ceph-mon[28998]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:41.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:40 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:41.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:40 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:41.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:40 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:42.321 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:41 smithi139 ceph-mon[32601]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:42.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:41 smithi110 ceph-mon[28998]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:42.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:41 smithi110 ceph-mon[39404]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:43.141 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.6 2024-09-11T08:31:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== 2024-09-11T08:31:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== == AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== ']' 2024-09-11T08:31:43.455 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:44.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:31:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:44 smithi110 ceph-mon[28998]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:44.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:44 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/903391328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:44.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:44 smithi110 ceph-mon[39404]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:44.404 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:44 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/903391328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:44.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:44 smithi139 ceph-mon[32601]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:44.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:44 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/903391328' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:46.139 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:31:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:45] "GET /metrics HTTP/1.1" 200 219772 "" "Prometheus/2.43.0" 2024-09-11T08:31:46.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:46 smithi110 ceph-mon[28998]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:46.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:46 smithi110 ceph-mon[39404]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:46.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:31:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:31:46.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:46 smithi139 ceph-mon[32601]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:47.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:47 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:47.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:47 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:47.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:47 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:48.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:48 smithi110 ceph-mon[28998]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:48.403 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:48 smithi110 ceph-mon[39404]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.6 2024-09-11T08:31:48.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:48 smithi139 ceph-mon[32601]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:48.771 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== 2024-09-11T08:31:48.771 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== == AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== ']' 2024-09-11T08:31:48.771 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:49.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:49 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1805210423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:49.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:49 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1805210423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:49.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:49 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1805210423' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:50.473 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:50 smithi139 ceph-mon[32601]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:50.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:50 smithi110 ceph-mon[28998]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:50.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:50 smithi110 ceph-mon[39404]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:52.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:52 smithi139 ceph-mon[32601]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:52.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:52 smithi110 ceph-mon[28998]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:52.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:52 smithi110 ceph-mon[39404]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:53.773 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.6 2024-09-11T08:31:54.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:31:53 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:31:54.091 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for osd.7 2024-09-11T08:31:54.092 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQBlVeFmEraODxAAC26HS7JppUkZRfaQtlC6zw== 2024-09-11T08:31:54.092 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQB6U+Fme1jeCBAA9dKEny1FgOVQV67p/wub6Q== == AQBlVeFmEraODxAAC26HS7JppUkZRfaQtlC6zw== ']' 2024-09-11T08:31:54.092 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:31:54.092 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for osd.7' 2024-09-11T08:31:54.092 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.7 2024-09-11T08:31:54.394 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== 2024-09-11T08:31:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== 2024-09-11T08:31:54.395 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key osd.7 2024-09-11T08:31:54.395 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:54 smithi110 ceph-mon[39404]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:54.395 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:54 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1811413434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:54.395 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:54 smithi110 ceph-mon[28998]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:54.395 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:54 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1811413434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:54.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:54 smithi139 ceph-mon[32601]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:54.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:54 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1811413434' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.6"}]: dispatch 2024-09-11T08:31:54.653 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key osd.7 on host 'smithi139' 2024-09-11T08:31:54.665 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== == AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== ']' 2024-09-11T08:31:54.665 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/258613808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.378 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-11T08:31:55.379 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-11T08:31:55.379 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2024-09-11T08:31:55.379 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:55 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/258613808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:31:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:55.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/258613808' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:55.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-11T08:31:55.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]: dispatch 2024-09-11T08:31:55.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "osd.7", "format": "json"}]': finished 2024-09-11T08:31:55.655 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:31:56.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:31:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:31:55] "GET /metrics HTTP/1.1" 200 219782 "" "Prometheus/2.43.0" 2024-09-11T08:31:56.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:31:56 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:31:56.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:56 smithi139 ceph-mon[32601]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:56.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:56 smithi139 ceph-mon[32601]: Schedule rotate-key daemon osd.7 2024-09-11T08:31:56.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:56 smithi139 ceph-mon[32601]: Rotating authentication key for osd.7 2024-09-11T08:31:56.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:56 smithi139 ceph-mon[32601]: Reconfiguring daemon osd.7 on smithi139 2024-09-11T08:31:56.572 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:56 smithi139 ceph-mon[32601]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[28998]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[28998]: Schedule rotate-key daemon osd.7 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[28998]: Rotating authentication key for osd.7 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[28998]: Reconfiguring daemon osd.7 on smithi139 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[28998]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[39404]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "osd.7", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[39404]: Schedule rotate-key daemon osd.7 2024-09-11T08:31:56.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[39404]: Rotating authentication key for osd.7 2024-09-11T08:31:56.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[39404]: Reconfiguring daemon osd.7 on smithi139 2024-09-11T08:31:56.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:56 smithi110 ceph-mon[39404]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:31:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:57 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:57 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:57 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:57 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:57 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:31:57.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:57.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:57 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:31:58.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:31:58 smithi139 ceph-mon[32601]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:58.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:31:58 smithi110 ceph-mon[28998]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:58.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:31:58 smithi110 ceph-mon[39404]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:31:59.667 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.7 2024-09-11T08:31:59.981 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== 2024-09-11T08:31:59.981 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== == AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== ']' 2024-09-11T08:31:59.981 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:32:00.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:00 smithi139 ceph-mon[32601]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:00.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:00 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4014186094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:00.652 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:00 smithi110 ceph-mon[28998]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:00.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:00 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4014186094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:00.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:00 smithi110 ceph-mon[39404]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:00.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:00 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4014186094' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:02.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:02 smithi139 ceph-mon[32601]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:02.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:02 smithi110 ceph-mon[28998]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:02.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:02 smithi110 ceph-mon[39404]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:04.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:03 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:32:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:04.571 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:04 smithi139 ceph-mon[32601]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:04.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:04 smithi110 ceph-mon[28998]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:04.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:04 smithi110 ceph-mon[39404]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:04.983 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.7 2024-09-11T08:32:05.298 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== 2024-09-11T08:32:05.298 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== == AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== ']' 2024-09-11T08:32:05.298 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:32:05.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:05 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1423007811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:05.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:05 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1423007811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:05.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:05 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1423007811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:06.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:05 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:32:05] "GET /metrics HTTP/1.1" 200 219782 "" "Prometheus/2.43.0" 2024-09-11T08:32:06.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:06 smithi110 ceph-mon[28998]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:06.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:06 smithi110 ceph-mon[39404]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:06.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:06 smithi139 ceph-mon[32601]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:06.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:32:06 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:32:07.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:07 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:07.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:07 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:07.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:07 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:08.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:08 smithi110 ceph-mon[28998]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:08.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:08 smithi110 ceph-mon[39404]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:08.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:08 smithi139 ceph-mon[32601]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:10.300 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.7 2024-09-11T08:32:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== 2024-09-11T08:32:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== == AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== ']' 2024-09-11T08:32:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:32:10.613 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:10 smithi110 ceph-mon[39404]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:10.613 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:10 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:10.613 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:10 smithi110 ceph-mon[28998]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:10.613 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:10 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:10.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:10 smithi139 ceph-mon[32601]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:10.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:10 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:11.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:11 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/278653668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:11.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:11 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/278653668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:11.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:11 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/278653668' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:12.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:12 smithi110 ceph-mon[28998]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:12.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:12 smithi110 ceph-mon[39404]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:12.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:12 smithi139 ceph-mon[32601]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:14.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:13 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:32:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:14.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:14 smithi110 ceph-mon[28998]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:14.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:14 smithi110 ceph-mon[39404]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:14.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:14 smithi139 ceph-mon[32601]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:15.614 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.7 2024-09-11T08:32:15.902 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:15 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:32:15] "GET /metrics HTTP/1.1" 200 219777 "" "Prometheus/2.43.0" 2024-09-11T08:32:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== 2024-09-11T08:32:15.929 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== == AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== ']' 2024-09-11T08:32:15.930 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:32:16.652 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:16 smithi110 ceph-mon[28998]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:16.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:16 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/3410479631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:16.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:16 smithi110 ceph-mon[39404]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:16.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:16 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/3410479631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:16.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:32:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:32:16.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:16 smithi139 ceph-mon[32601]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:16.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:16 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/3410479631' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:17.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:17 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:17.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:17 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:17.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:17 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:18.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:18 smithi110 ceph-mon[28998]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:18.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:18 smithi110 ceph-mon[39404]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:18.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:18 smithi139 ceph-mon[32601]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:20.624 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:20 smithi139 ceph-mon[32601]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:20.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:20 smithi110 ceph-mon[28998]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:20.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:20 smithi110 ceph-mon[39404]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-11T08:32:20.931 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key osd.7 2024-09-11T08:32:21.244 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for mgr.y 2024-09-11T08:32:21.245 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQB6VeFmCtLsKxAA3jqQSvGNu8vFFWLRdTM4SA== 2024-09-11T08:32:21.245 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQCRU+Fm3mqqKhAAn7PsTb2+pUcQxXbPbgVjeg== == AQB6VeFmCtLsKxAA3jqQSvGNu8vFFWLRdTM4SA== ']' 2024-09-11T08:32:21.245 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:32:21.245 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for mgr.y' 2024-09-11T08:32:21.245 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key mgr.y 2024-09-11T08:32:21.552 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQB7UuFmx1v8GRAATpUkamblIVyfy7J+D37c5A== 2024-09-11T08:32:21.552 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQB7UuFmx1v8GRAATpUkamblIVyfy7J+D37c5A== 2024-09-11T08:32:21.552 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key mgr.y 2024-09-11T08:32:21.552 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:21 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2081499676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:21.553 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:21 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2081499676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:21.640 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:21 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2081499676' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "osd.7"}]: dispatch 2024-09-11T08:32:21.807 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key mgr.y on host 'smithi110' 2024-09-11T08:32:21.819 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQB7UuFmx1v8GRAATpUkamblIVyfy7J+D37c5A== == AQB7UuFmx1v8GRAATpUkamblIVyfy7J+D37c5A== ']' 2024-09-11T08:32:21.819 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/2728063684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:22.555 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/2728063684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:22.556 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:22.557 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:22 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:22.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-11T08:32:22.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/2728063684' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-11T08:32:22.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:22.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:22 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:23.583 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[28998]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:32:23.583 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[28998]: Schedule rotate-key daemon mgr.y 2024-09-11T08:32:23.584 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[28998]: Rotating authentication key for mgr.y 2024-09-11T08:32:23.584 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[28998]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:32:23.584 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:23.584 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:23.584 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[39404]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:32:23.584 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[39404]: Schedule rotate-key daemon mgr.y 2024-09-11T08:32:23.584 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[39404]: Rotating authentication key for mgr.y 2024-09-11T08:32:23.585 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[39404]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:32:23.585 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:23.585 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:23.585 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.439+0000 7f361c66b640 -1 mgr.server reply reply (13) Permission denied access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication 2024-09-11T08:32:23.585 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.440+0000 7f360e5cf640 -1 log_channel(cephadm) log [ERR] : Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-y/keyring', '-n', 'mgr.y', 'tell', 'mgr.y', 'rotate-key', '-i', '-']: 2024-09-11T08:32:23.434+0000 7fb5f959a640 1 -- 172.21.15.110:0/3184583622 <== mon.2 v2:172.21.15.110:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7fb5e4003710 con 0x7fb5fc104fc0 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.434+0000 7fb5f959a640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 1177758842162294449 != 18374858748799134293 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- 172.21.15.110:0/3184583622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5fc104fc0 msgr2=0x7fb5fc1053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 --2- 172.21.15.110:0/3184583622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5fc104fc0 0x7fb5fc1053f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec004950 tx=0x7fb5ec009da0 comp rx=0 tx=0).stop 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- 172.21.15.110:0/3184583622 shutdown_connections 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 --2- 172.21.15.110:0/3184583622 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc106ce0 0x7fb5fc10d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 --2- 172.21.15.110:0/3184583622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5fc106330 0x7fb5fc1067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 --2- 172.21.15.110:0/3184583622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5fc104fc0 0x7fb5fc1053f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- 172.21.15.110:0/3184583622 >> 172.21.15.110:0/3184583622 conn(0x7fb5fc06b9d0 msgr2=0x7fb5fc06de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- 172.21.15.110:0/3184583622 shutdown_connections 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- 172.21.15.110:0/3184583622 wait complete. 2024-09-11T08:32:23.586 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 Processor -- start 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- start start 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5fc104fc0 0x7fb5fc068cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc106330 0x7fb5fc067310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5fc106ce0 0x7fb5fc067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc067ef0 con 0x7fb5fc106ce0 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc068060 con 0x7fb5fc106330 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.435+0000 7fb600bbf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc11f720 con 0x7fb5fc104fc0 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5fc104fc0 0x7fb5fc068cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5fc104fc0 0x7fb5fc068cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43348/0 (socket says 172.21.15.110:43348) 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 -- 172.21.15.110:0/1580364611 learned_addr learned my addr 172.21.15.110:0/1580364611 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fad9d640 1 --2- 172.21.15.110:0/1580364611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5fc106ce0 0x7fb5fc067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T08:32:23.587 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 -- 172.21.15.110:0/1580364611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc106330 msgr2=0x7fb5fc067310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 --2- 172.21.15.110:0/1580364611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb5fc106330 0x7fb5fc067310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 -- 172.21.15.110:0/1580364611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5fc106ce0 msgr2=0x7fb5fc067850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 --2- 172.21.15.110:0/1580364611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5fc106ce0 0x7fb5fc067850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 -- 172.21.15.110:0/1580364611 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0005040 con 0x7fb5fc104fc0 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fad9d640 1 --2- 172.21.15.110:0/1580364611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5fc106ce0 0x7fb5fc067850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5fa59c640 1 --2- 172.21.15.110:0/1580364611 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5fc104fc0 0x7fb5fc068cb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0047f0 tx=0x7fb5ec009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec032e90 con 0x7fb5fc104fc0 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7fb5e40032d0 con 0x7fb5fc104fc0 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(38 keys) v1 ==== 1752+0+0 (secure 0 0 0) 0x7fb5ec009820 con 0x7fb5fc104fc0 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb600bbf640 1 -- 172.21.15.110:0/1580364611 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ec005040 con 0x7fb5fc104fc0 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec031e90 con 0x7fb5fc104fc0 2024-09-11T08:32:23.588 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb600bbf640 1 -- 172.21.15.110:0/1580364611 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc11fc80 con 0x7fb5fc104fc0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 <== mon.2 v2:172.21.15.110:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7fb5fc11fc80 con 0x7fb5fc104fc0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.436+0000 7fb5eaffd640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 1177758842162294449 != 18374858748799134293 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.438+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 <== mon.2 v2:172.21.15.110:3301/0 5 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fb5ec04c020 con 0x7fb5fc104fc0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.438+0000 7fb5eaffd640 1 --2- 172.21.15.110:0/1580364611 >> v2:172.21.15.110:6800/264388069 conn(0x7fb5e406f1d0 0x7fb5e4071680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.438+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 --> v2:172.21.15.110:6800/264388069 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5e4071d50 con 0x7fb5e406f1d0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.438+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 <== mon.2 v2:172.21.15.110:3301/0 6 ==== osd_map(64..64 src has 1..64) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fb5ec0af740 con 0x7fb5fc104fc0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.438+0000 7fb5f9d9b640 1 --2- 172.21.15.110:0/1580364611 >> v2:172.21.15.110:6800/264388069 conn(0x7fb5e406f1d0 0x7fb5e4071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.438+0000 7fb5f9d9b640 1 --2- 172.21.15.110:0/1580364611 >> v2:172.21.15.110:6800/264388069 conn(0x7fb5e406f1d0 0x7fb5e4071680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc065210 tx=0x7fb5f0009da0 comp rx=0 tx=0).ready entity=mgr.14574 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.439+0000 7fb5eaffd640 1 -- 172.21.15.110:0/1580364611 <== mgr.14574 v2:172.21.15.110:6800/264388069 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (secure 0 0 0) 0x7fb5e4071d50 con 0x7fb5e406f1d0 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: Error EACCES: problem getting command descriptions from mgr.y 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.589 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.451+0000 7f363307a640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-11T08:32:23.590 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setuser ceph since I am not root 2024-09-11T08:32:23.590 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: ignoring --setgroup ceph since I am not root 2024-09-11T08:32:23.590 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mgr[29229]: -- 172.21.15.110:0/4274581328 <== mon.0 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a2a77b8340 con 0x55a2a77c4c00 2024-09-11T08:32:23.590 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-mgr[29229]: cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 1177758842162294449 != 18374858748799134293 2024-09-11T08:32:23.590 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.583+0000 7f8de2587ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:32:23.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-mon[32601]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.y", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:32:23.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-mon[32601]: Schedule rotate-key daemon mgr.y 2024-09-11T08:32:23.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-mon[32601]: Rotating authentication key for mgr.y 2024-09-11T08:32:23.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-mon[32601]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:32:23.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:23.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' 2024-09-11T08:32:23.816 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:23] ENGINE Bus STOPPING 2024-09-11T08:32:23.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.637+0000 7f8de2587ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:32:23.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.703+0000 7f8de2587ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:32:23.903 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.825+0000 7f8de2587ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:32:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:32:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:32:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:23] ENGINE Bus STOPPED 2024-09-11T08:32:24.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:23] ENGINE Bus STARTING 2024-09-11T08:32:24.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:23] ENGINE Serving on http://:::9283 2024-09-11T08:32:24.072 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:23] ENGINE Bus STARTED 2024-09-11T08:32:24.224 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:23 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:23.963+0000 7f8de2587ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:32:24.225 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:24.224+0000 7f8de2587ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:32:24.368 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-11T08:32:24.368 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: Activating manager daemon x 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: osdmap e65: 8 total, 8 up, 8 in 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: mgrmap e23: x(active, starting, since 0.00603651s) 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:32:24.369 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:32:24.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:24.287+0000 7f8de2587ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:32:24.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:24.470+0000 7f8de2587ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:32:24.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-11T08:32:24.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-11T08:32:24.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: Activating manager daemon x 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: osdmap e65: 8 total, 8 up, 8 in 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: mgrmap e23: x(active, starting, since 0.00603651s) 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:32:24.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: Manager daemon x is now available 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:24.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.14574 172.21.15.110:0/4090371208' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: Activating manager daemon x 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: osdmap e65: 8 total, 8 up, 8 in 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.14574 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: mgrmap e23: x(active, starting, since 0.00603651s) 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:32:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: Manager daemon x is now available 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:24 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:24.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:32:24.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:32:24.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:32:24.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:32:24.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:32:24.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:32:24.747 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: Manager daemon x is now available 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:24.748 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:24 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:25.454 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:25.097+0000 7f80279f2640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:32:25] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: Traceback (most recent call last): 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: self._connections.run(self.expiration_interval) 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: self._run(expiration_interval) 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: s = self.context.wrap_socket( 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:25.455 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: return self.sslsocket_class._create( 2024-09-11T08:32:25.456 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:25.456 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: self.do_handshake() 2024-09-11T08:32:25.456 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:25.456 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: self._sslobj.do_handshake() 2024-09-11T08:32:25.456 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:25.456 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-mon[28998]: mgrmap e24: x(active, since 1.00873s) 2024-09-11T08:32:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-mon[28998]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:25.653 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:25.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-mon[39404]: mgrmap e24: x(active, since 1.00873s) 2024-09-11T08:32:25.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-mon[39404]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:25.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:25.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:25.400+0000 7f8de2587ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:32:25.654 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:25 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:25.477+0000 7f8de2587ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:32:25.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-mon[32601]: mgrmap e24: x(active, since 1.00873s) 2024-09-11T08:32:25.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-mon[32601]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:25.754 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:25 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:26.521 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.212+0000 7f8de2587ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:32:26.521 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.333+0000 7f8de2587ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:32:26.521 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.459+0000 7f8de2587ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:32:26.521 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:24] ENGINE Bus STARTING 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:25] ENGINE Serving on https://172.21.15.139:7150 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:25] ENGINE Bus STARTED 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:25] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:26.522 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:32:26.523 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:26.524 INFO:journalctl@ceph.mon.a.smithi110.stdout: self.do_handshake() 2024-09-11T08:32:26.524 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:26.524 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:32:26.524 INFO:journalctl@ceph.mon.a.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:26.524 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[28998]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:26.524 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[28998]: mgrmap e25: x(active, since 2s) 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:24] ENGINE Bus STARTING 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:25] ENGINE Serving on https://172.21.15.139:7150 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:25] ENGINE Bus STARTED 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:25] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:32:26.525 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:26.526 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:32:26.526 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:26.526 INFO:journalctl@ceph.mon.c.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:26.526 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:26.526 INFO:journalctl@ceph.mon.c.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:26.526 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: self.do_handshake() 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:26.527 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[39404]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:26.528 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-mon[39404]: mgrmap e25: x(active, since 2s) 2024-09-11T08:32:26.780 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.519+0000 7f8de2587ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:32:26.781 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.651+0000 7f8de2587ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:32:26.781 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.719+0000 7f8de2587ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:32:26.781 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.779+0000 7f8de2587ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:32:26.821 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key mgr.y 2024-09-11T08:32:26.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:32:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:32:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:26 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:24] ENGINE Bus STARTING 2024-09-11T08:32:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:26 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:25] ENGINE Serving on https://172.21.15.139:7150 2024-09-11T08:32:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:26 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:25] ENGINE Bus STARTED 2024-09-11T08:32:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:26 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:25] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:26.822 INFO:journalctl@ceph.mon.b.smithi139.stdout: Traceback (most recent call last): 2024-09-11T08:32:26.823 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:26.823 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:32:26.823 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:26.823 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._run(expiration_interval) 2024-09-11T08:32:26.823 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:26.824 INFO:journalctl@ceph.mon.b.smithi139.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:26.824 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:26.824 INFO:journalctl@ceph.mon.b.smithi139.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:26.824 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:26.824 INFO:journalctl@ceph.mon.b.smithi139.stdout: s = self.context.wrap_socket( 2024-09-11T08:32:26.824 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:26.824 INFO:journalctl@ceph.mon.b.smithi139.stdout: return self.sslsocket_class._create( 2024-09-11T08:32:26.825 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:26.825 INFO:journalctl@ceph.mon.b.smithi139.stdout: self.do_handshake() 2024-09-11T08:32:26.825 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:26.825 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._sslobj.do_handshake() 2024-09-11T08:32:26.825 INFO:journalctl@ceph.mon.b.smithi139.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:26.825 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:26 smithi139 ceph-mon[32601]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:26.826 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:26 smithi139 ceph-mon[32601]: mgrmap e25: x(active, since 2s) 2024-09-11T08:32:27.075 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:26 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:26.936+0000 7f8de2587ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:32:27.075 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:27.004+0000 7f8de2587ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:32:27.075 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:27.073+0000 7f8de2587ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:32:27.160 INFO:teuthology.orchestra.run.smithi110.stdout:rotating key for mgr.x 2024-09-11T08:32:27.160 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQCVVeFmlnmoNBAAxwdEDlym68L8R8PgWhZ0HA== 2024-09-11T08:32:27.161 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQB7UuFmx1v8GRAATpUkamblIVyfy7J+D37c5A== == AQCVVeFmlnmoNBAAxwdEDlym68L8R8PgWhZ0HA== ']' 2024-09-11T08:32:27.161 INFO:teuthology.orchestra.run.smithi110.stderr:+ for f in osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 mgr.y mgr.x 2024-09-11T08:32:27.161 INFO:teuthology.orchestra.run.smithi110.stderr:+ echo 'rotating key for mgr.x' 2024-09-11T08:32:27.161 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key mgr.x 2024-09-11T08:32:27.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:27.166+0000 7f8de2587ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:32:27.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:27.228+0000 7f8de2587ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:32:27.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:27.291+0000 7f8de2587ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:32:27.486 INFO:teuthology.orchestra.run.smithi110.stderr:+ K=AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== 2024-09-11T08:32:27.487 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== 2024-09-11T08:32:27.487 INFO:teuthology.orchestra.run.smithi110.stderr:+ ceph orch daemon rotate-key mgr.x 2024-09-11T08:32:27.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:27 smithi139 ceph-mon[32601]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:27.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:27 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:27.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:27 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:27.735 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:27 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1848196254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-11T08:32:27.779 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled to rotate-key mgr.x on host 'smithi139' 2024-09-11T08:32:27.779 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[28998]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:27.779 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:27.779 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:27.779 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1848196254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-11T08:32:27.779 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[39404]: from='client.24391 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-11T08:32:27.780 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:27.780 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:27.780 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:27 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1848196254' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.y"}]: dispatch 2024-09-11T08:32:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== == AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== ']' 2024-09-11T08:32:27.789 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:32:28.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:28.078+0000 7f8de2587ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:32:28.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:32:28] ENGINE Bus STARTING 2024-09-11T08:32:28.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: CherryPy Checker: 2024-09-11T08:32:28.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: The Application mounted at '' has an empty config. 2024-09-11T08:32:28.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: 2024-09-11T08:32:28.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:32:28] ENGINE Serving on http://:::9283 2024-09-11T08:32:28.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[29225]: [11/Sep/2024:08:32:28] ENGINE Bus STARTED 2024-09-11T08:32:28.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: mgrmap e26: x(active, since 4s) 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/1456420811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: Standby manager daemon y started 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:28 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: mgrmap e26: x(active, since 4s) 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/1456420811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.903 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: Standby manager daemon y started 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:28.904 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: mgrmap e26: x(active, since 4s) 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/1456420811' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.906 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: Standby manager daemon y started 2024-09-11T08:32:28.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:28.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:28.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-11T08:32:28.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.110:0/3371316804' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:28.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:28.907 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:28 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi110 to 3248M 2024-09-11T08:32:29.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:32:29.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: Schedule rotate-key daemon mgr.x 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:29.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:29.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: mgrmap e27: x(active, since 5s), standbys: y 2024-09-11T08:32:29.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:29 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi110 to 3248M 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: Schedule rotate-key daemon mgr.x 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: mgrmap e27: x(active, since 5s), standbys: y 2024-09-11T08:32:29.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi110 to 3248M 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "rotate-key", "name": "mgr.x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: Schedule rotate-key daemon mgr.x 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-11T08:32:29.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-11T08:32:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: mgrmap e27: x(active, since 5s), standbys: y 2024-09-11T08:32:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:29 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Adjusting osd_memory_target on smithi139 to 3248M 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:30.823 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:30 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Adjusting osd_memory_target on smithi139 to 3248M 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.903 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:30.904 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Adjusting osd_memory_target on smithi139 to 3248M 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi110:/etc/ceph/ceph.conf 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi139:/etc/ceph/ceph.conf 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-11T08:32:30.905 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: Updating smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]: dispatch 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.y", "format": "json"}]': finished 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:30 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:31.799 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[28998]: Rotating authentication key for mgr.y 2024-09-11T08:32:31.799 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[28998]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:32:31.799 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:31.799 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:31.800 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[39404]: Rotating authentication key for mgr.y 2024-09-11T08:32:31.800 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[39404]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:32:31.800 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:31.800 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:31 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:31.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:31 smithi139 ceph-mon[32601]: Rotating authentication key for mgr.y 2024-09-11T08:32:31.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:31 smithi139 ceph-mon[32601]: Reconfiguring daemon mgr.y on smithi110 2024-09-11T08:32:31.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:31 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:31.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:31 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:32.115 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:31 smithi110 systemd[1]: Stopping Ceph mgr.y for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:32.377 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 podman[81089]: 2024-09-11 08:32:32.115326473 +0000 UTC m=+0.173764591 container died 6dce3c0c4cd04a90db33a4b29b582d632f47c112e67a6bfc842922848c04e757 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-11T08:32:32.652 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:32 smithi110 ceph-mon[28998]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:32:32.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:32 smithi110 ceph-mon[39404]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:32:32.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 podman[81089]: 2024-09-11 08:32:32.484093426 +0000 UTC m=+0.542531533 container cleanup 6dce3c0c4cd04a90db33a4b29b582d632f47c112e67a6bfc842922848c04e757 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:32:32.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 bash[81089]: ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y 2024-09-11T08:32:32.791 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key mgr.x 2024-09-11T08:32:32.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:32 smithi139 ceph-mon[32601]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-11T08:32:33.118 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== 2024-09-11T08:32:33.118 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== == AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== ']' 2024-09-11T08:32:33.118 INFO:teuthology.orchestra.run.smithi110.stderr:+ sleep 5 2024-09-11T08:32:33.118 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 podman[81100]: 2024-09-11 08:32:32.752767529 +0000 UTC m=+0.636932859 container remove 6dce3c0c4cd04a90db33a4b29b582d632f47c112e67a6bfc842922848c04e757 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-09-11T08:32:33.118 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:32:33.118 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-11T08:32:33.118 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 systemd[1]: Stopped Ceph mgr.y for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:32:33.118 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service: Consumed 45.192s CPU time. 2024-09-11T08:32:33.118 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:32 smithi110 systemd[1]: Starting Ceph mgr.y for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:33.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 podman[81215]: 2024-09-11 08:32:33.032424433 +0000 UTC m=+0.015584953 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9 2024-09-11T08:32:33.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 podman[81215]: 2024-09-11 08:32:33.148894934 +0000 UTC m=+0.132055454 container create 936039b832659e3b208d47ab3347f62f5f47d451215daa7798acb6dd6a46ff3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-11T08:32:33.664 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/26057038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:33.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/26057038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:33.664 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 podman[81215]: 2024-09-11 08:32:33.423823439 +0000 UTC m=+0.406983959 container init 936039b832659e3b208d47ab3347f62f5f47d451215daa7798acb6dd6a46ff3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 podman[81215]: 2024-09-11 08:32:33.426668798 +0000 UTC m=+0.409829319 container start 936039b832659e3b208d47ab3347f62f5f47d451215daa7798acb6dd6a46ff3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0) 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-mgr[81237]: -- 172.21.15.110:0/3147988137 <== mon.0 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa3616a340 con 0x55fa36177000 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-mgr[81237]: cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 1826110854127870748 != 18374858748799134293 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 bash[81215]: 936039b832659e3b208d47ab3347f62f5f47d451215daa7798acb6dd6a46ff3f 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 systemd[1]: Started Ceph mgr.y for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:33.542+0000 7fe3f5f12ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:33.598+0000 7fe3f5f12ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:32:33.665 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:33.663+0000 7fe3f5f12ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:32:33.815 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:33 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/26057038' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:33.921 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:33.786+0000 7fe3f5f12ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:32:33.921 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:33 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:33.920+0000 7fe3f5f12ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:32:34.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:32:33] "GET /metrics HTTP/1.1" 200 34933 "" "Prometheus/2.43.0" 2024-09-11T08:32:34.545 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:34.177+0000 7fe3f5f12ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:32:34.546 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:34.237+0000 7fe3f5f12ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:32:34.546 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:34.398+0000 7fe3f5f12ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:32:34.805 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]': finished 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:32:34.806 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]': finished 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:34.807 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:34 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]: dispatch 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create-pending", "entity": "mgr.x", "format": "json"}]': finished 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:34.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-11T08:32:34.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-11T08:32:34.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:34 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-11T08:32:35.214 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.066+0000 7f805d25d640 -1 mgr.server reply reply (13) Permission denied access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication 2024-09-11T08:32:35.214 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.068+0000 7f805aa58640 -1 log_channel(cephadm) log [ERR] : Non-zero return from ['ceph', '-k', '/var/lib/ceph/mgr/ceph-x/keyring', '-n', 'mgr.x', 'tell', 'mgr.x', 'rotate-key', '-i', '-']: 2024-09-11T08:32:35.061+0000 7f4880dd9640 1 -- 172.21.15.139:0/2775656653 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7f4868003710 con 0x7f487c104710 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.061+0000 7f4880dd9640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 13097094899758079627 != 18374858748799134293 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 -- 172.21.15.139:0/2775656653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f487c104710 msgr2=0x7f487c106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 --2- 172.21.15.139:0/2775656653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f487c104710 0x7f487c106af0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4878004950 tx=0x7f4878009da0 comp rx=0 tx=0).stop 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 -- 172.21.15.139:0/2775656653 shutdown_connections 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 --2- 172.21.15.139:0/2775656653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f487c109b70 0x7f487c10c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 --2- 172.21.15.139:0/2775656653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487c1070c0 0x7f487c1094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 --2- 172.21.15.139:0/2775656653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f487c104710 0x7f487c106af0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 -- 172.21.15.139:0/2775656653 >> 172.21.15.139:0/2775656653 conn(0x7f487c06a8f0 msgr2=0x7f487c06cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 -- 172.21.15.139:0/2775656653 shutdown_connections 2024-09-11T08:32:35.215 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 -- 172.21.15.139:0/2775656653 wait complete. 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 Processor -- start 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.062+0000 7f488403f640 1 -- start start 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487c104710 0x7f487c115790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f487c1070c0 0x7f487c115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f487c109b70 0x7f487c11cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c11d2f0 con 0x7f487c104710 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c11d460 con 0x7f487c109b70 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487c11d620 con 0x7f487c1070c0 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f487c109b70 0x7f487c11cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f487c109b70 0x7f487c11cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:51954/0 (socket says 172.21.15.139:51954) 2024-09-11T08:32:35.216 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 -- 172.21.15.139:0/2799492522 learned_addr learned my addr 172.21.15.139:0/2799492522 (peer_addr_for_me v2:172.21.15.139:0/0) 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f4881ddb640 1 --2- 172.21.15.139:0/2799492522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487c104710 0x7f487c115790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48815da640 1 --2- 172.21.15.139:0/2799492522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f487c1070c0 0x7f487c115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 -- 172.21.15.139:0/2799492522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f487c1070c0 msgr2=0x7f487c115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 --2- 172.21.15.139:0/2799492522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f487c1070c0 0x7f487c115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 -- 172.21.15.139:0/2799492522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487c104710 msgr2=0x7f487c115790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 --2- 172.21.15.139:0/2799492522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487c104710 0x7f487c115790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 -- 172.21.15.139:0/2799492522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878005040 con 0x7f487c109b70 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48815da640 1 --2- 172.21.15.139:0/2799492522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f487c1070c0 0x7f487c115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48825dc640 1 --2- 172.21.15.139:0/2799492522 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f487c109b70 0x7f487c11cdb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f487400bb80 tx=0x7f4874009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T08:32:35.217 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48740092e0 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- auth(proto 2 2 bytes epoch 0) v1 -- 0x7f48680032d0 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(38 keys) v1 ==== 1752+0+0 (secure 0 0 0) 0x7f4874015470 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 -- 172.21.15.139:0/2799492522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c11d900 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874007d50 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f488403f640 1 -- 172.21.15.139:0/2799492522 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c11ddf0 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.063+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x7f487c11ddf0 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.064+0000 7f48727fc640 0 cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 13097094899758079627 != 18374858748799134293 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.065+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 <== mon.1 v2:172.21.15.139:3300/0 5 ==== mgrmap(e 27) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f4874012070 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.065+0000 7f48727fc640 1 --2- 172.21.15.139:0/2799492522 >> v2:172.21.15.139:6816/2798891763 conn(0x7f486806f1c0 0x7f4868071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.065+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 --> v2:172.21.15.139:6816/2798891763 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4868071d40 con 0x7f486806f1c0 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.065+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 <== mon.1 v2:172.21.15.139:3300/0 6 ==== osd_map(65..65 src has 1..65) v4 ==== 6499+0+0 (secure 0 0 0) 0x7f487408ebf0 con 0x7f487c109b70 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.065+0000 7f4881ddb640 1 --2- 172.21.15.139:0/2799492522 >> v2:172.21.15.139:6816/2798891763 conn(0x7f486806f1c0 0x7f4868071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.066+0000 7f4881ddb640 1 --2- 172.21.15.139:0/2799492522 >> v2:172.21.15.139:6816/2798891763 conn(0x7f486806f1c0 0x7f4868071670 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f487802a4a0 tx=0x7f4878009da0 comp rx=0 tx=0).ready entity=mgr.24436 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T08:32:35.218 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.066+0000 7f48727fc640 1 -- 172.21.15.139:0/2799492522 <== mgr.24436 v2:172.21.15.139:6816/2798891763 1 ==== command_reply(tid 0: -13 access denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication) v1 ==== 134+0+0 (secure 0 0 0) 0x7f4868071d40 con 0x7f486806f1c0 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: Error EACCES: problem getting command descriptions from mgr.x 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.081+0000 7f807fe84640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ignoring --setuser ceph since I am not root 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ignoring --setgroup ceph since I am not root 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mgr[34437]: -- 172.21.15.139:0/2819766810 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5593bc3dc340 con 0x5593bc3e8800 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mgr[34437]: cephx client: could not set rotating key: decode_decrypt failed. error:bad magic in decode_decrypt, 13097094899758079627 != 18374858748799134293 2024-09-11T08:32:35.219 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.213+0000 7f4abdaa4ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:32:35.356 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:35.288+0000 7fe3f5f12ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:32:35.549 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.269+0000 7f4abdaa4ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:32:35.549 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.334+0000 7f4abdaa4ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:32:35.549 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.456+0000 7f4abdaa4ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:35.354+0000 7fe3f5f12ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: Rotating authentication key for mgr.x 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: Reconfiguring daemon mgr.x on smithi139 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-11T08:32:35.653 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: Activating manager daemon y 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: osdmap e66: 8 total, 8 up, 8 in 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[39404]: mgrmap e28: y(active, starting, since 0.00624223s) 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: Rotating authentication key for mgr.x 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: Reconfiguring daemon mgr.x on smithi139 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: Activating manager daemon y 2024-09-11T08:32:35.654 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: osdmap e66: 8 total, 8 up, 8 in 2024-09-11T08:32:35.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-11T08:32:35.655 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:35 smithi110 ceph-mon[28998]: mgrmap e28: y(active, starting, since 0.00624223s) 2024-09-11T08:32:35.821 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.592+0000 7f4abdaa4ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:32:35.821 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: Rotating authentication key for mgr.x 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: Reconfiguring daemon mgr.x on smithi139 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: from='mgr.24436 172.21.15.139:0/617063585' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: Activating manager daemon y 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: osdmap e66: 8 total, 8 up, 8 in 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: from='mgr.24436 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-11T08:32:35.822 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-mon[32601]: mgrmap e28: y(active, starting, since 0.00624223s) 2024-09-11T08:32:36.321 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.848+0000 7f4abdaa4ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:32:36.321 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:35.908+0000 7f4abdaa4ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:32:36.321 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:36.068+0000 7f4abdaa4ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:32:36.362 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.072+0000 7fe3f5f12ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:32:36.362 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.183+0000 7fe3f5f12ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:32:36.362 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.303+0000 7fe3f5f12ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:32:36.363 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.361+0000 7fe3f5f12ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:32:36.614 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.487+0000 7fe3f5f12ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:32:36.614 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.554+0000 7fe3f5f12ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:32:36.614 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.613+0000 7fe3f5f12ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:32:36.875 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.746+0000 7fe3f5f12ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:32:36.876 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.810+0000 7fe3f5f12ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:32:36.876 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.874+0000 7fe3f5f12ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:32:37.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:36 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:36.954+0000 7fe3f5f12ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:32:37.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:37.011+0000 7fe3f5f12ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:32:37.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:37.067+0000 7fe3f5f12ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:32:37.321 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:36.907+0000 7f4abdaa4ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:32:37.321 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:36.971+0000 7f4abdaa4ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:32:37.966 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:37.673+0000 7f4abdaa4ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T08:32:37.966 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:37.784+0000 7f4abdaa4ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:32:37.966 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:37.907+0000 7f4abdaa4ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:32:37.966 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:37.966+0000 7f4abdaa4ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: Active manager daemon y restarted 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: Activating manager daemon y 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: osdmap e67: 8 total, 8 up, 8 in 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: mgrmap e29: y(active, starting, since 0.00669178s) 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:32:37.967 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:37.968 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-11T08:32:37.969 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:37 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:38.121 INFO:teuthology.orchestra.run.smithi110.stderr:++ ceph auth get-key mgr.x 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: Active manager daemon y restarted 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: Activating manager daemon y 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: osdmap e67: 8 total, 8 up, 8 in 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: mgrmap e29: y(active, starting, since 0.00669178s) 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:32:38.122 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:32:38.123 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:32:38.124 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:38.124 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-11T08:32:38.124 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:38.124 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:37.808+0000 7fe3f5f12ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:32:38.124 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:37] ENGINE Bus STARTING 2024-09-11T08:32:38.124 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: CherryPy Checker: 2024-09-11T08:32:38.124 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: The Application mounted at '' has an empty config. 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:37] ENGINE Serving on http://:::9283 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:37] ENGINE Bus STARTED 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:37] ENGINE Bus STOPPING 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:37] ENGINE Bus STOPPED 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:38] ENGINE Bus STARTING 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: Active manager daemon y restarted 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: Activating manager daemon y 2024-09-11T08:32:38.125 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: osdmap e67: 8 total, 8 up, 8 in 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: mgrmap e29: y(active, starting, since 0.00669178s) 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-11T08:32:38.126 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-11T08:32:38.127 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:37 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:38.220 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.093+0000 7f4abdaa4ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T08:32:38.220 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.159+0000 7f4abdaa4ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:32:38.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:38] ENGINE Serving on http://:::9283 2024-09-11T08:32:38.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:38] ENGINE Bus STARTED 2024-09-11T08:32:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:+ NK=AQChVeFm7a8hIRAA/hQ2fLR16fzaUK9qEkrG3Q== 2024-09-11T08:32:38.461 INFO:teuthology.orchestra.run.smithi110.stderr:+ '[' AQDpUuFmX0BJIBAAKvBKx5veOvVLN/bFrzEE2A== == AQChVeFm7a8hIRAA/hQ2fLR16fzaUK9qEkrG3Q== ']' 2024-09-11T08:32:38.499 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.220+0000 7f4abdaa4ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:32:38.499 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.362+0000 7f4abdaa4ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:32:38.499 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.430+0000 7f4abdaa4ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:32:38.499 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.499+0000 7f4abdaa4ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:32:38.821 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.586+0000 7f4abdaa4ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:32:38.821 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.653+0000 7f4abdaa4ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:32:38.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:38.719+0000 7f4abdaa4ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:32:39.085 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[28998]: Manager daemon y is now available 2024-09-11T08:32:39.085 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:39.085 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[28998]: from='client.? 172.21.15.110:0/4012503362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[39404]: Manager daemon y is now available 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:39.086 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:38 smithi110 ceph-mon[39404]: from='client.? 172.21.15.110:0/4012503362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:39.083+0000 7fe398a93640 -1 log_channel(cephadm) log [ERR] : [11/Sep/2024:08:32:39] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: Traceback (most recent call last): 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: self._connections.run(self.expiration_interval) 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: self._run(expiration_interval) 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:39.087 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:39.137 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-11T08:32:39.146 INFO:tasks.cephadm:Teardown begin 2024-09-11T08:32:39.147 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:39.174 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:32:39.197 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug there is no tcmu-runner data available 2024-09-11T08:32:39.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-mon[32601]: Manager daemon y is now available 2024-09-11T08:32:39.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-11T08:32:39.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-11T08:32:39.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-11T08:32:39.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-11T08:32:39.198 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:38 smithi139 ceph-mon[32601]: from='client.? 172.21.15.110:0/4012503362' entity='client.admin' cmd=[{"prefix": "auth get-key", "entity": "mgr.x"}]: dispatch 2024-09-11T08:32:39.202 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-11T08:32:39.202 DEBUG:teuthology.orchestra.run.smithi110:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-11T08:32:39.230 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-11T08:32:39.258 INFO:tasks.cephadm:Stopping all daemons... 2024-09-11T08:32:39.258 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-11T08:32:39.258 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 systemd[1]: Stopping Ceph mon.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: s = self.context.wrap_socket( 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: return self.sslsocket_class._create( 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: self.do_handshake() 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: self._sslobj.do_handshake() 2024-09-11T08:32:39.403 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:39.404 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:39.821 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:39.514+0000 7f4abdaa4ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:32:39.821 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:39] ENGINE Bus STARTING 2024-09-11T08:32:39.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: CherryPy Checker: 2024-09-11T08:32:39.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: The Application mounted at '' has an empty config. 2024-09-11T08:32:39.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: 2024-09-11T08:32:39.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:39] ENGINE Serving on http://:::9283 2024-09-11T08:32:39.822 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: [11/Sep/2024:08:32:39] ENGINE Bus STARTED 2024-09-11T08:32:40.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:40] ENGINE Bus STOPPING 2024-09-11T08:32:40.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: mgrmap e30: y(active, since 1.06321s) 2024-09-11T08:32:40.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:38] ENGINE Bus STARTING 2024-09-11T08:32:40.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:39] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:32:40.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:39] ENGINE Bus STARTED 2024-09-11T08:32:40.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: [11/Sep/2024:08:32:39] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:40.153 INFO:journalctl@ceph.mon.c.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:32:40.154 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout: self.do_handshake() 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: Standby manager daemon x started 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[39404]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:40.155 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: mgrmap e30: y(active, since 1.06321s) 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:38] ENGINE Bus STARTING 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:39] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:39] ENGINE Bus STARTED 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: [11/Sep/2024:08:32:39] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: Traceback (most recent call last): 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._run(expiration_interval) 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:40.156 INFO:journalctl@ceph.mon.a.smithi110.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: s = self.context.wrap_socket( 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: return self.sslsocket_class._create( 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: self.do_handshake() 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: self._sslobj.do_handshake() 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: Standby manager daemon x started 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:40.157 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:32:40.158 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:39 smithi110 ceph-mon[28998]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:40.158 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a[28994]: 2024-09-11T08:32:40.042+0000 7f12c075d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:40.158 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a[28994]: 2024-09-11T08:32:40.042+0000 7f12c075d640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-11T08:32:40.162 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: mgrmap e30: y(active, since 1.06321s) 2024-09-11T08:32:40.162 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:38] ENGINE Bus STARTING 2024-09-11T08:32:40.162 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:39] ENGINE Serving on https://172.21.15.110:7150 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:39] ENGINE Bus STARTED 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: [11/Sep/2024:08:32:39] ENGINE Error in HTTPServer.serve 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: Traceback (most recent call last): 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._connections.run(self.expiration_interval) 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._run(expiration_interval) 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-11T08:32:40.163 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: s = self.context.wrap_socket( 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: return self.sslsocket_class._create( 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: self.do_handshake() 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: self._sslobj.do_handshake() 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: from='mgr.15435 172.21.15.110:0/980892227' entity='mgr.y' 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: Standby manager daemon x started 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-11T08:32:40.164 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:39 smithi139 ceph-mon[32601]: from='mgr.? 172.21.15.139:0/1542097435' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-11T08:32:40.511 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:40 smithi110 podman[81750]: 2024-09-11 08:32:40.237518527 +0000 UTC m=+0.284426811 container died cf0c495a244ec64c4e0297d7d16f56794ea933956c7d81a96c7761cc264875e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-11T08:32:40.512 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:32:40.512 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:40] ENGINE Bus STOPPED 2024-09-11T08:32:40.512 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:40] ENGINE Bus STARTING 2024-09-11T08:32:40.512 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:40] ENGINE Serving on http://:::9283 2024-09-11T08:32:40.512 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:40 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:40] ENGINE Bus STARTED 2024-09-11T08:32:40.878 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:40 smithi110 podman[81750]: 2024-09-11 08:32:40.602699626 +0000 UTC m=+0.649607916 container cleanup cf0c495a244ec64c4e0297d7d16f56794ea933956c7d81a96c7761cc264875e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD) 2024-09-11T08:32:40.879 INFO:journalctl@ceph.mon.a.smithi110.stdout:Sep 11 08:32:40 smithi110 bash[81750]: ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-a 2024-09-11T08:32:41.042 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.a.service' 2024-09-11T08:32:41.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:41.110 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-11T08:32:41.110 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-11T08:32:41.110 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.c 2024-09-11T08:32:41.402 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:41 smithi110 systemd[1]: Stopping Ceph mon.c for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:42.153 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:41 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:41] ENGINE Bus STOPPING 2024-09-11T08:32:42.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:41 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-c[39400]: 2024-09-11T08:32:41.876+0000 7f1d85974640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:42.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:41 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-c[39400]: 2024-09-11T08:32:41.876+0000 7f1d85974640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-11T08:32:42.438 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:42 smithi110 podman[82091]: 2024-09-11 08:32:42.179537582 +0000 UTC m=+0.742894607 container died ce5888c4df4d68e20fd132bf7f585538f243b2e79571ced0874f1c0ec96fcaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-11T08:32:42.438 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-11T08:32:42.438 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:42] ENGINE Bus STOPPED 2024-09-11T08:32:42.438 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:42] ENGINE Bus STARTING 2024-09-11T08:32:42.439 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:42] ENGINE Serving on http://:::9283 2024-09-11T08:32:42.439 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:42 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: [11/Sep/2024:08:32:42] ENGINE Bus STARTED 2024-09-11T08:32:42.697 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:42 smithi110 podman[82091]: 2024-09-11 08:32:42.538192597 +0000 UTC m=+1.101549664 container cleanup ce5888c4df4d68e20fd132bf7f585538f243b2e79571ced0874f1c0ec96fcaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-c, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-11T08:32:42.697 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:42 smithi110 bash[82091]: ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-c 2024-09-11T08:32:43.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 11 08:32:42 smithi110 podman[82152]: 2024-09-11 08:32:42.697795458 +0000 UTC m=+0.689522071 container remove ce5888c4df4d68e20fd132bf7f585538f243b2e79571ced0874f1c0ec96fcaa2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:32:43.302 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.c.service' 2024-09-11T08:32:43.340 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:43.340 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-11T08:32:43.340 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-11T08:32:43.340 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.b 2024-09-11T08:32:43.652 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:43 smithi139 systemd[1]: Stopping Ceph mon.b for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:44.071 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x[34433]: ::ffff:172.21.15.139 - - [11/Sep/2024:08:32:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-11T08:32:44.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b[32585]: 2024-09-11T08:32:43.652+0000 7f7a6b90a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:44.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b[32585]: 2024-09-11T08:32:43.652+0000 7f7a6b90a640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-11T08:32:44.072 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:43 smithi139 podman[69623]: 2024-09-11 08:32:43.866699157 +0000 UTC m=+0.329094104 container died 648681997ea9acf3e3afaed75b4053b282bbf59666a93cd905b8f01553fdb718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-11T08:32:44.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:44 smithi139 podman[69623]: 2024-09-11 08:32:44.227127432 +0000 UTC m=+0.689522376 container cleanup 648681997ea9acf3e3afaed75b4053b282bbf59666a93cd905b8f01553fdb718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:32:44.511 INFO:journalctl@ceph.mon.b.smithi139.stdout:Sep 11 08:32:44 smithi139 bash[69623]: ceph-2850e098-7016-11ef-bcea-c7b262605968-mon-b 2024-09-11T08:32:44.646 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mon.b.service' 2024-09-11T08:32:44.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:44.696 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-11T08:32:44.697 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-11T08:32:44.697 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y 2024-09-11T08:32:45.011 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:44 smithi110 systemd[1]: Stopping Ceph mgr.y for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:45.284 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:45 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y[81233]: 2024-09-11T08:32:45.010+0000 7fe3f1ea7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:45.284 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:45 smithi110 podman[82291]: 2024-09-11 08:32:45.041625335 +0000 UTC m=+0.151153328 container died 936039b832659e3b208d47ab3347f62f5f47d451215daa7798acb6dd6a46ff3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-11T08:32:45.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:45 smithi110 podman[82291]: 2024-09-11 08:32:45.42812284 +0000 UTC m=+0.537650832 container cleanup 936039b832659e3b208d47ab3347f62f5f47d451215daa7798acb6dd6a46ff3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d0e6828a2016d48cf25ad84064e50742bb1c39b9, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:32:45.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Sep 11 08:32:45 smithi110 bash[82291]: ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-y 2024-09-11T08:32:45.835 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-11T08:32:45.872 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:45.872 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-11T08:32:45.873 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-11T08:32:45.873 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.x 2024-09-11T08:32:46.203 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:45 smithi139 systemd[1]: Stopping Ceph mgr.x for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:46.499 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:46 smithi139 podman[69748]: 2024-09-11 08:32:46.203764856 +0000 UTC m=+0.141003602 container died c0dfedff762315a0e4c76a77f513c31b2341b2e8733d0a8b8d1dcb1cd3c7f15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-11T08:32:46.821 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:46 smithi139 podman[69748]: 2024-09-11 08:32:46.582685553 +0000 UTC m=+0.519924298 container cleanup c0dfedff762315a0e4c76a77f513c31b2341b2e8733d0a8b8d1dcb1cd3c7f15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-11T08:32:46.821 INFO:journalctl@ceph.mgr.x.smithi139.stdout:Sep 11 08:32:46 smithi139 bash[69748]: ceph-2850e098-7016-11ef-bcea-c7b262605968-mgr-x 2024-09-11T08:32:47.011 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-11T08:32:47.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:47.047 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-11T08:32:47.047 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-11T08:32:47.047 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.0 2024-09-11T08:32:47.332 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:47 smithi110 systemd[1]: Stopping Ceph osd.0 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:47.652 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:47 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0[43237]: 2024-09-11T08:32:47.332+0000 7f65e1331640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:47.653 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:47 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0[43237]: 2024-09-11T08:32:47.332+0000 7f65e1331640 -1 osd.0 67 *** Got signal Terminated *** 2024-09-11T08:32:47.653 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:47 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0[43237]: 2024-09-11T08:32:47.332+0000 7f65e1331640 -1 osd.0 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:32:53.139 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:52 smithi110 podman[82414]: 2024-09-11 08:32:52.822198162 +0000 UTC m=+5.579566396 container died e2add389dd6d50f77073abd05873fbda5b9ae93a268a9a45097e627dd971a541 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-11T08:32:53.403 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 podman[82414]: 2024-09-11 08:32:53.140454173 +0000 UTC m=+5.897822406 container cleanup e2add389dd6d50f77073abd05873fbda5b9ae93a268a9a45097e627dd971a541 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-11T08:32:53.403 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 bash[82414]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0 2024-09-11T08:32:53.403 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 podman[82431]: 2024-09-11 08:32:53.358707115 +0000 UTC m=+0.528153230 container remove e2add389dd6d50f77073abd05873fbda5b9ae93a268a9a45097e627dd971a541 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-11T08:32:53.986 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 podman[82517]: 2024-09-11 08:32:53.579259547 +0000 UTC m=+0.015825120 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:32:53.986 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 podman[82517]: 2024-09-11 08:32:53.69306534 +0000 UTC m=+0.129630912 container create 1be8fd5ebdc879bd54961181dcda19788a3629ec88fd7042e72cafcff6b53e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0-deactivate, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-11T08:32:53.986 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 podman[82517]: 2024-09-11 08:32:53.912049856 +0000 UTC m=+0.348615442 container init 1be8fd5ebdc879bd54961181dcda19788a3629ec88fd7042e72cafcff6b53e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-09-11T08:32:53.987 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 podman[82517]: 2024-09-11 08:32:53.915374711 +0000 UTC m=+0.351940284 container start 1be8fd5ebdc879bd54961181dcda19788a3629ec88fd7042e72cafcff6b53e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-11T08:32:53.987 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:53 smithi110 podman[82517]: 2024-09-11 08:32:53.986998777 +0000 UTC m=+0.423564348 container attach 1be8fd5ebdc879bd54961181dcda19788a3629ec88fd7042e72cafcff6b53e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:32:54.381 INFO:journalctl@ceph.osd.0.smithi110.stdout:Sep 11 08:32:54 smithi110 podman[82517]: 2024-09-11 08:32:54.093299792 +0000 UTC m=+0.529865372 container died 1be8fd5ebdc879bd54961181dcda19788a3629ec88fd7042e72cafcff6b53e05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2) 2024-09-11T08:32:54.667 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.0.service' 2024-09-11T08:32:54.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:32:54.706 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-11T08:32:54.706 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-11T08:32:54.707 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.1 2024-09-11T08:32:54.902 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:32:54 smithi110 systemd[1]: Stopping Ceph osd.1 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:32:55.402 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:32:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1[48826]: 2024-09-11T08:32:55.019+0000 7fa11b359640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:32:55.403 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:32:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1[48826]: 2024-09-11T08:32:55.019+0000 7fa11b359640 -1 osd.1 67 *** Got signal Terminated *** 2024-09-11T08:32:55.403 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:32:55 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1[48826]: 2024-09-11T08:32:55.019+0000 7fa11b359640 -1 osd.1 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:00.653 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:00 smithi110 podman[82618]: 2024-09-11 08:33:00.396235535 +0000 UTC m=+5.473275169 container died acf291d8e2378391ce9c584f22d11756871064ce6dbeaad8215e0056bb716cf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:33:00.927 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:00 smithi110 podman[82618]: 2024-09-11 08:33:00.735506531 +0000 UTC m=+5.812546162 container cleanup acf291d8e2378391ce9c584f22d11756871064ce6dbeaad8215e0056bb716cf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:00.928 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:00 smithi110 bash[82618]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1 2024-09-11T08:33:01.342 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:00 smithi110 podman[82637]: 2024-09-11 08:33:00.928465834 +0000 UTC m=+0.524979167 container remove acf291d8e2378391ce9c584f22d11756871064ce6dbeaad8215e0056bb716cf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-09-11T08:33:01.596 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:01 smithi110 podman[82724]: 2024-09-11 08:33:01.245417023 +0000 UTC m=+0.015869618 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:01.596 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:01 smithi110 podman[82724]: 2024-09-11 08:33:01.354327364 +0000 UTC m=+0.124780005 container create a14b2146c17064d0a1c143e6fc348a57cf9589a8760335c939b94c5f6237fe87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-11T08:33:01.596 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:01 smithi110 podman[82724]: 2024-09-11 08:33:01.596233911 +0000 UTC m=+0.366686506 container init a14b2146c17064d0a1c143e6fc348a57cf9589a8760335c939b94c5f6237fe87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:01.903 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:01 smithi110 podman[82724]: 2024-09-11 08:33:01.599481906 +0000 UTC m=+0.369934503 container start a14b2146c17064d0a1c143e6fc348a57cf9589a8760335c939b94c5f6237fe87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-11T08:33:01.904 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:01 smithi110 podman[82724]: 2024-09-11 08:33:01.689928526 +0000 UTC m=+0.460381132 container attach a14b2146c17064d0a1c143e6fc348a57cf9589a8760335c939b94c5f6237fe87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-11T08:33:01.904 INFO:journalctl@ceph.osd.1.smithi110.stdout:Sep 11 08:33:01 smithi110 podman[82724]: 2024-09-11 08:33:01.772934915 +0000 UTC m=+0.543387513 container died a14b2146c17064d0a1c143e6fc348a57cf9589a8760335c939b94c5f6237fe87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-11T08:33:02.429 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.1.service' 2024-09-11T08:33:02.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:02.471 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-11T08:33:02.472 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-11T08:33:02.472 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.2 2024-09-11T08:33:02.652 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:02 smithi110 systemd[1]: Stopping Ceph osd.2 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:33:03.153 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:02 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2[54806]: 2024-09-11T08:33:02.788+0000 7f5c28e50640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:03.153 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:02 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2[54806]: 2024-09-11T08:33:02.788+0000 7f5c28e50640 -1 osd.2 67 *** Got signal Terminated *** 2024-09-11T08:33:03.153 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:02 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2[54806]: 2024-09-11T08:33:02.788+0000 7f5c28e50640 -1 osd.2 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:08.403 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:08 smithi110 podman[82829]: 2024-09-11 08:33:08.137950033 +0000 UTC m=+5.441271677 container died d9506e518aa63423bcac6857c9067f6922ead9bcba901eba7f9cb35c518f8486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-11T08:33:08.655 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:08 smithi110 podman[82829]: 2024-09-11 08:33:08.454533418 +0000 UTC m=+5.757855098 container cleanup d9506e518aa63423bcac6857c9067f6922ead9bcba901eba7f9cb35c518f8486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-11T08:33:08.655 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:08 smithi110 bash[82829]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2 2024-09-11T08:33:09.073 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:08 smithi110 podman[82844]: 2024-09-11 08:33:08.655654322 +0000 UTC m=+0.506674826 container remove d9506e518aa63423bcac6857c9067f6922ead9bcba901eba7f9cb35c518f8486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-11T08:33:09.341 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:09 smithi110 podman[82933]: 2024-09-11 08:33:08.976311001 +0000 UTC m=+0.015626591 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:09.342 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:09 smithi110 podman[82933]: 2024-09-11 08:33:09.089853065 +0000 UTC m=+0.129168642 container create 7b504ac1a2f1e45e1f6bd8f034f8bba81347e427e16e576e696cd50f7d270e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-11T08:33:09.342 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:09 smithi110 podman[82933]: 2024-09-11 08:33:09.342375454 +0000 UTC m=+0.381691030 container init 7b504ac1a2f1e45e1f6bd8f034f8bba81347e427e16e576e696cd50f7d270e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-11T08:33:09.653 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:09 smithi110 podman[82933]: 2024-09-11 08:33:09.345704393 +0000 UTC m=+0.385019974 container start 7b504ac1a2f1e45e1f6bd8f034f8bba81347e427e16e576e696cd50f7d270e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:09.653 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:09 smithi110 podman[82933]: 2024-09-11 08:33:09.425747024 +0000 UTC m=+0.465062617 container attach 7b504ac1a2f1e45e1f6bd8f034f8bba81347e427e16e576e696cd50f7d270e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-11T08:33:09.653 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 11 08:33:09 smithi110 podman[82933]: 2024-09-11 08:33:09.522241266 +0000 UTC m=+0.561556852 container died 7b504ac1a2f1e45e1f6bd8f034f8bba81347e427e16e576e696cd50f7d270e6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-11T08:33:10.102 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.2.service' 2024-09-11T08:33:10.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:10.140 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-11T08:33:10.140 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-11T08:33:10.141 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.3 2024-09-11T08:33:10.360 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:10 smithi110 systemd[1]: Stopping Ceph osd.3 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:33:10.653 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:10 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3[60913]: 2024-09-11T08:33:10.448+0000 7f2d9b45f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:10.653 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:10 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3[60913]: 2024-09-11T08:33:10.448+0000 7f2d9b45f640 -1 osd.3 67 *** Got signal Terminated *** 2024-09-11T08:33:10.653 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:10 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3[60913]: 2024-09-11T08:33:10.448+0000 7f2d9b45f640 -1 osd.3 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:14.903 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:14 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3[60913]: 2024-09-11T08:33:14.454+0000 7f2d96274640 -1 osd.3 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:49.566676+0000 front 2024-09-11T08:32:49.566847+0000 (oldest deadline 2024-09-11T08:33:13.666803+0000) 2024-09-11T08:33:15.071 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:14 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:14.591+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:16.071 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:15 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:15.620+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:16.124 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:15 smithi110 podman[83036]: 2024-09-11 08:33:15.855447662 +0000 UTC m=+5.490277451 container died 413622f354745827c941399a01e49fe87a57b25c095cd83c47574ec0998bce8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-11T08:33:16.383 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:16 smithi110 podman[83036]: 2024-09-11 08:33:16.19870612 +0000 UTC m=+5.833535906 container cleanup 413622f354745827c941399a01e49fe87a57b25c095cd83c47574ec0998bce8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:16.383 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:16 smithi110 bash[83036]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3 2024-09-11T08:33:16.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:16.311+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:16.652 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:16 smithi110 podman[83064]: 2024-09-11 08:33:16.383188206 +0000 UTC m=+0.523182795 container remove 413622f354745827c941399a01e49fe87a57b25c095cd83c47574ec0998bce8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:17.071 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:16.610+0000 7f5230ee5640 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:52.284021+0000 front 2024-09-11T08:32:52.284101+0000 (oldest deadline 2024-09-11T08:33:15.784025+0000) 2024-09-11T08:33:17.071 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:16 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:16.616+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:17.086 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:16 smithi110 podman[83154]: 2024-09-11 08:33:16.704396287 +0000 UTC m=+0.015684718 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:17.086 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:16 smithi110 podman[83154]: 2024-09-11 08:33:16.808958048 +0000 UTC m=+0.120246475 container create 56106a69744f95e2558599664a6f3903554310ada1f367fdab324295d9b255eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:17.086 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:17 smithi110 podman[83154]: 2024-09-11 08:33:17.003207037 +0000 UTC m=+0.314495462 container init 56106a69744f95e2558599664a6f3903554310ada1f367fdab324295d9b255eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:17.086 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:17 smithi110 podman[83154]: 2024-09-11 08:33:17.006445359 +0000 UTC m=+0.317733787 container start 56106a69744f95e2558599664a6f3903554310ada1f367fdab324295d9b255eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:17.086 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:17 smithi110 podman[83154]: 2024-09-11 08:33:17.086308472 +0000 UTC m=+0.397596896 container attach 56106a69744f95e2558599664a6f3903554310ada1f367fdab324295d9b255eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-11T08:33:17.403 INFO:journalctl@ceph.osd.3.smithi110.stdout:Sep 11 08:33:17 smithi110 podman[83154]: 2024-09-11 08:33:17.183193519 +0000 UTC m=+0.494481946 container died 56106a69744f95e2558599664a6f3903554310ada1f367fdab324295d9b255eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS) 2024-09-11T08:33:17.628 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:17 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:17.575+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:17.628 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:17 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:17.358+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:17.721 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.3.service' 2024-09-11T08:33:17.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:17.762 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-11T08:33:17.762 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-11T08:33:17.762 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.4 2024-09-11T08:33:18.042 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:17 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:17.628+0000 7f5230ee5640 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:52.284021+0000 front 2024-09-11T08:32:52.284101+0000 (oldest deadline 2024-09-11T08:33:15.784025+0000) 2024-09-11T08:33:18.043 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:17 smithi139 systemd[1]: Stopping Ceph osd.4 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:33:18.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:18 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:18.111+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:18.321 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:18 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:18.042+0000 7f52348cd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:18.321 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:18 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:18.042+0000 7f52348cd640 -1 osd.4 67 *** Got signal Terminated *** 2024-09-11T08:33:18.322 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:18 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:18.042+0000 7f52348cd640 -1 osd.4 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:18.821 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:18 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:18.629+0000 7f5230ee5640 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:52.284021+0000 front 2024-09-11T08:32:52.284101+0000 (oldest deadline 2024-09-11T08:33:15.784025+0000) 2024-09-11T08:33:18.821 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:18 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:18.617+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:18.822 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:18 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:18.399+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:19.395 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:19 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:19.092+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:19.821 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:19 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:19.617+0000 7f5230ee5640 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:52.284021+0000 front 2024-09-11T08:32:52.284101+0000 (oldest deadline 2024-09-11T08:33:15.784025+0000) 2024-09-11T08:33:19.821 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:19 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:19.636+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:19.822 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:19 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:19.395+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:20.427 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:20 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:20.079+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:20.821 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:20 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:20.612+0000 7f5230ee5640 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:52.284021+0000 front 2024-09-11T08:32:52.284101+0000 (oldest deadline 2024-09-11T08:33:15.784025+0000) 2024-09-11T08:33:20.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:20 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:20.598+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:20.822 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:20 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:20.427+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:21.410 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:21 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:21.088+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:21.821 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:21 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:21.640+0000 7f5230ee5640 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:52.284021+0000 front 2024-09-11T08:32:52.284101+0000 (oldest deadline 2024-09-11T08:33:15.784025+0000) 2024-09-11T08:33:21.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:21 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:21.624+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:21.822 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:21 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:21.410+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:21.822 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:21 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:21.410+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:22.422 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:22 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:22.135+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:22.687 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:22 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:22.588+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:22.687 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:22 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:22.422+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:22.687 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:22 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:22.422+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:23.071 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:22 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4[37382]: 2024-09-11T08:33:22.687+0000 7f5230ee5640 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:52.284021+0000 front 2024-09-11T08:32:52.284101+0000 (oldest deadline 2024-09-11T08:33:15.784025+0000) 2024-09-11T08:33:23.432 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:23.091+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:23.776 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:23 smithi139 podman[69883]: 2024-09-11 08:33:23.531219004 +0000 UTC m=+5.572955943 container died 6dbe3088d9f02342df26a6f1b5dc6799ce4ec3bf60190653587cd82fc3b64542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-11T08:33:23.776 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:23.627+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:23.777 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:23.432+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:23.777 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:23 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:23.432+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:24.051 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:23 smithi139 podman[69883]: 2024-09-11 08:33:23.883940075 +0000 UTC m=+5.925677018 container cleanup 6dbe3088d9f02342df26a6f1b5dc6799ce4ec3bf60190653587cd82fc3b64542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-11T08:33:24.051 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:23 smithi139 bash[69883]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4 2024-09-11T08:33:24.322 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:24 smithi139 podman[69899]: 2024-09-11 08:33:24.05167655 +0000 UTC m=+0.515960056 container remove 6dbe3088d9f02342df26a6f1b5dc6799ce4ec3bf60190653587cd82fc3b64542 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-11T08:33:24.322 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:24 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:24.112+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:24.322 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:24 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:24.112+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:24.713 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:24 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:24.614+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:24.713 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:24 smithi139 podman[69988]: 2024-09-11 08:33:24.394608147 +0000 UTC m=+0.015852459 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:24.713 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:24 smithi139 podman[69988]: 2024-09-11 08:33:24.494233883 +0000 UTC m=+0.115478181 container create 2ae0bc593da69d9277cd66c7b3df0639ec60fef3ac1a5e01e73c6536fc356d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-11T08:33:24.714 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:24 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:24.422+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:24.714 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:24 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:24.423+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:25.072 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:24 smithi139 podman[69988]: 2024-09-11 08:33:24.71403999 +0000 UTC m=+0.335284289 container init 2ae0bc593da69d9277cd66c7b3df0639ec60fef3ac1a5e01e73c6536fc356d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:25.072 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:24 smithi139 podman[69988]: 2024-09-11 08:33:24.717714801 +0000 UTC m=+0.338959108 container start 2ae0bc593da69d9277cd66c7b3df0639ec60fef3ac1a5e01e73c6536fc356d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-09-11T08:33:25.072 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:24 smithi139 podman[69988]: 2024-09-11 08:33:24.839244661 +0000 UTC m=+0.460488967 container attach 2ae0bc593da69d9277cd66c7b3df0639ec60fef3ac1a5e01e73c6536fc356d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-11T08:33:25.072 INFO:journalctl@ceph.osd.4.smithi139.stdout:Sep 11 08:33:24 smithi139 podman[69988]: 2024-09-11 08:33:24.886920436 +0000 UTC m=+0.508164736 container died 2ae0bc593da69d9277cd66c7b3df0639ec60fef3ac1a5e01e73c6536fc356d43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-11T08:33:25.417 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:25.088+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:25.417 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:25.088+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:25.542 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.4.service' 2024-09-11T08:33:25.585 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:25.585 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-11T08:33:25.585 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-11T08:33:25.586 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.5 2024-09-11T08:33:25.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:25.417+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:25.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:25.417+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:25.821 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:25.615+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:25.821 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:25.615+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:59.945269+0000 front 2024-09-11T08:32:59.945206+0000 (oldest deadline 2024-09-11T08:33:25.245025+0000) 2024-09-11T08:33:25.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:25 smithi139 systemd[1]: Stopping Ceph osd.5 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:33:26.114 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:25.963+0000 7f9b09fb5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:26.114 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:25.963+0000 7f9b09fb5640 -1 osd.5 67 *** Got signal Terminated *** 2024-09-11T08:33:26.114 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:25 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:25.963+0000 7f9b09fb5640 -1 osd.5 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:26.397 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:26.114+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:26.398 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:26.114+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:26.795 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:26.603+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:26.795 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:26.603+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:59.945269+0000 front 2024-09-11T08:32:59.945206+0000 (oldest deadline 2024-09-11T08:33:25.245025+0000) 2024-09-11T08:33:26.795 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:26.397+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:26.795 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:26 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:26.397+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:27.389 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:27 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:27.088+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:27.389 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:27 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:27.088+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:27.643 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:27 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:27.389+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:27.643 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:27 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:27.389+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:28.044 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:27 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:27.643+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:28.044 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:27 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:27.643+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:59.945269+0000 front 2024-09-11T08:32:59.945206+0000 (oldest deadline 2024-09-11T08:33:25.245025+0000) 2024-09-11T08:33:28.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:28.044+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:28.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:28.044+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:28.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:28.427+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:28.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:28.427+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:28.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:28.627+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:28.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:28.627+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:59.945269+0000 front 2024-09-11T08:32:59.945206+0000 (oldest deadline 2024-09-11T08:33:25.245025+0000) 2024-09-11T08:33:28.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:28.627+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.245562+0000 front 2024-09-11T08:33:05.245504+0000 (oldest deadline 2024-09-11T08:33:28.145349+0000) 2024-09-11T08:33:29.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:28.997+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:29.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:28 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:28.997+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:29.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:29 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:29.476+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:29.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:29 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:29.476+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:29.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:29 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:29.609+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:29.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:29 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:29.609+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:59.945269+0000 front 2024-09-11T08:32:59.945206+0000 (oldest deadline 2024-09-11T08:33:25.245025+0000) 2024-09-11T08:33:29.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:29 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:29.609+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.245562+0000 front 2024-09-11T08:33:05.245504+0000 (oldest deadline 2024-09-11T08:33:28.145349+0000) 2024-09-11T08:33:30.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:30.032+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:30.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:30.032+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:30.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:30.435+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:30.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:30.435+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:30.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:30.435+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:30.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:30.639+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:48.744288+0000 front 2024-09-11T08:32:48.744215+0000 (oldest deadline 2024-09-11T08:33:14.044166+0000) 2024-09-11T08:33:30.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:30.639+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:59.945269+0000 front 2024-09-11T08:32:59.945206+0000 (oldest deadline 2024-09-11T08:33:25.245025+0000) 2024-09-11T08:33:30.822 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:30 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5[43395]: 2024-09-11T08:33:30.639+0000 7f9b055cb640 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.245562+0000 front 2024-09-11T08:33:05.245504+0000 (oldest deadline 2024-09-11T08:33:28.145349+0000) 2024-09-11T08:33:31.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:31.054+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:31.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:31.054+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:31.706 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:31 smithi139 podman[70089]: 2024-09-11 08:33:31.436966662 +0000 UTC m=+5.591452315 container died 337873bc5ef241551bdcedfd121fc241a87fcf787ab42fa73f91310961a04a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:33:31.707 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:31.476+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:31.707 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:31.476+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:31.707 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:31 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:31.476+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:32.065 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:31 smithi139 podman[70089]: 2024-09-11 08:33:31.814578584 +0000 UTC m=+5.969064230 container cleanup 337873bc5ef241551bdcedfd121fc241a87fcf787ab42fa73f91310961a04a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:32.065 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:31 smithi139 bash[70089]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5 2024-09-11T08:33:32.321 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:32 smithi139 podman[70115]: 2024-09-11 08:33:32.065589864 +0000 UTC m=+0.620347691 container remove 337873bc5ef241551bdcedfd121fc241a87fcf787ab42fa73f91310961a04a84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-11T08:33:32.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:32 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:32.095+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:32.321 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:32 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:32.095+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:32.322 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:32 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:32.095+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:32.659 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:32 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:32.435+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:32.659 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:32 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:32.435+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:32.660 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:32 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:32.435+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:32.660 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:32 smithi139 podman[70203]: 2024-09-11 08:33:32.28522622 +0000 UTC m=+0.015698785 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:32.660 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:32 smithi139 podman[70203]: 2024-09-11 08:33:32.38306221 +0000 UTC m=+0.113534784 container create 84103938ee0dfa8c250e168b10783a26a489b14ccda60755caef63034c45a83b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731) 2024-09-11T08:33:32.660 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:32 smithi139 podman[70203]: 2024-09-11 08:33:32.577109467 +0000 UTC m=+0.307582032 container init 84103938ee0dfa8c250e168b10783a26a489b14ccda60755caef63034c45a83b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-11T08:33:32.660 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:32 smithi139 podman[70203]: 2024-09-11 08:33:32.580424014 +0000 UTC m=+0.310896580 container start 84103938ee0dfa8c250e168b10783a26a489b14ccda60755caef63034c45a83b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:32.660 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:32 smithi139 podman[70203]: 2024-09-11 08:33:32.660222205 +0000 UTC m=+0.390694781 container attach 84103938ee0dfa8c250e168b10783a26a489b14ccda60755caef63034c45a83b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:33.057 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:32 smithi139 podman[70203]: 2024-09-11 08:33:32.759604057 +0000 UTC m=+0.490076623 container died 84103938ee0dfa8c250e168b10783a26a489b14ccda60755caef63034c45a83b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-11T08:33:33.299 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.5.service' 2024-09-11T08:33:33.321 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:33 smithi139 podman[70226]: 2024-09-11 08:33:33.273194869 +0000 UTC m=+0.507519841 container remove 84103938ee0dfa8c250e168b10783a26a489b14ccda60755caef63034c45a83b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, ceph=True) 2024-09-11T08:33:33.321 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:33 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.5.service: Deactivated successfully. 2024-09-11T08:33:33.321 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:33 smithi139 systemd[1]: Stopped Ceph osd.5 for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:33:33.321 INFO:journalctl@ceph.osd.5.smithi139.stdout:Sep 11 08:33:33 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.5.service: Consumed 4.900s CPU time. 2024-09-11T08:33:33.322 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:33.094+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:33.322 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:33.094+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:33.322 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:33.094+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:33.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:33.339 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-11T08:33:33.339 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-11T08:33:33.339 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.6 2024-09-11T08:33:33.667 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:33.390+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:33.668 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:33.390+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:33.668 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:33.390+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:33.668 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:33 smithi139 systemd[1]: Stopping Ceph osd.6 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:33:33.668 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:33.666+0000 7fbdf8a85640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:33.668 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:33.666+0000 7fbdf8a85640 -1 osd.6 67 *** Got signal Terminated *** 2024-09-11T08:33:33.668 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:33 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:33.666+0000 7fbdf8a85640 -1 osd.6 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:34.401 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:34 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:34.073+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:34.401 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:34 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:34.073+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:34.401 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:34 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:34.073+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:34.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:34 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:34.400+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:34.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:34 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:34.400+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:34.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:34 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:34.400+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:35.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:35.361+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:35.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:35.361+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:35.572 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:35.361+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:35.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:35.118+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:35.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:35.118+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:35.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:35 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:35.118+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:36.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:36.375+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:36.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:36.375+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:36.572 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:36.375+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:36.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:36.136+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:36.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:36.136+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:36.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:36 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:36.136+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:37.423 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:37.129+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:37.423 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:37.129+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:37.423 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:37.129+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:37.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:37.423+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:37.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:37.423+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:37.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:37 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:37.423+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:38.429 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:38.106+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:38.429 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:38.106+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:38.430 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:38.106+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:38.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:38.428+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.341548+0000 front 2024-09-11T08:32:51.341718+0000 (oldest deadline 2024-09-11T08:33:15.441531+0000) 2024-09-11T08:33:38.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:38.428+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:55.442144+0000 front 2024-09-11T08:32:55.442050+0000 (oldest deadline 2024-09-11T08:33:20.742089+0000) 2024-09-11T08:33:38.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:38.428+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:05.343314+0000 front 2024-09-11T08:33:05.343302+0000 (oldest deadline 2024-09-11T08:33:30.043246+0000) 2024-09-11T08:33:38.821 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:38 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6[49324]: 2024-09-11T08:33:38.428+0000 7fbdf389a640 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:13.444627+0000 front 2024-09-11T08:33:13.444574+0000 (oldest deadline 2024-09-11T08:33:38.144544+0000) 2024-09-11T08:33:39.259 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:38 smithi139 podman[70306]: 2024-09-11 08:33:38.981216454 +0000 UTC m=+5.417339793 container died cb515397300ed825e9324c41ce127646687916477a066ca3057ffc8cca9bcdfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-11T08:33:39.259 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:39.064+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:39.259 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:39.064+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:39.259 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:39 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:39.064+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:39.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:39 smithi139 podman[70306]: 2024-09-11 08:33:39.367214458 +0000 UTC m=+5.803337794 container cleanup cb515397300ed825e9324c41ce127646687916477a066ca3057ffc8cca9bcdfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-11T08:33:39.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:39 smithi139 bash[70306]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6 2024-09-11T08:33:39.918 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:39 smithi139 podman[70322]: 2024-09-11 08:33:39.60990587 +0000 UTC m=+0.624633815 container remove cb515397300ed825e9324c41ce127646687916477a066ca3057ffc8cca9bcdfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:40.187 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:40.093+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:40.187 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:40.093+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:40.188 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:40.093+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:40.188 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:40 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:40.093+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:14.702081+0000 front 2024-09-11T08:33:14.702037+0000 (oldest deadline 2024-09-11T08:33:40.001909+0000) 2024-09-11T08:33:40.188 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:39 smithi139 podman[70411]: 2024-09-11 08:33:39.919006637 +0000 UTC m=+0.098995034 container create c69a6bc91df032ff3478087fd87d0520b558ee3cd8410e7f1c0da9fa15f7afa9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:40.188 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:39 smithi139 podman[70411]: 2024-09-11 08:33:39.835831878 +0000 UTC m=+0.015820275 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:40.188 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:40 smithi139 podman[70411]: 2024-09-11 08:33:40.104638851 +0000 UTC m=+0.284627248 container init c69a6bc91df032ff3478087fd87d0520b558ee3cd8410e7f1c0da9fa15f7afa9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, maintainer=Guillaume Abrioux ) 2024-09-11T08:33:40.188 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:40 smithi139 podman[70411]: 2024-09-11 08:33:40.10824395 +0000 UTC m=+0.288232351 container start c69a6bc91df032ff3478087fd87d0520b558ee3cd8410e7f1c0da9fa15f7afa9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-11T08:33:40.188 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:40 smithi139 podman[70411]: 2024-09-11 08:33:40.188062755 +0000 UTC m=+0.368051168 container attach c69a6bc91df032ff3478087fd87d0520b558ee3cd8410e7f1c0da9fa15f7afa9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.license=GPLv2) 2024-09-11T08:33:40.571 INFO:journalctl@ceph.osd.6.smithi139.stdout:Sep 11 08:33:40 smithi139 podman[70411]: 2024-09-11 08:33:40.279704855 +0000 UTC m=+0.459693255 container died c69a6bc91df032ff3478087fd87d0520b558ee3cd8410e7f1c0da9fa15f7afa9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-6-deactivate, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-11T08:33:40.863 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.6.service' 2024-09-11T08:33:40.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:40.901 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-11T08:33:40.901 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-11T08:33:40.902 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.7 2024-09-11T08:33:41.100 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:40 smithi139 systemd[1]: Stopping Ceph osd.7 for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:33:41.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:41.099+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:41.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:41.099+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:41.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:41.099+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:41.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:41.099+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:14.702081+0000 front 2024-09-11T08:33:14.702037+0000 (oldest deadline 2024-09-11T08:33:40.001909+0000) 2024-09-11T08:33:41.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:41.201+0000 7f9b839a5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:41.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:41.201+0000 7f9b839a5640 -1 osd.7 67 *** Got signal Terminated *** 2024-09-11T08:33:41.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:41 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:41.201+0000 7f9b839a5640 -1 osd.7 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-11T08:33:41.903 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:33:41 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:33:41.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi110.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.110:8443: connect: connection refused" 2024-09-11T08:33:41.903 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:33:41 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:33:41.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-09-11T08:33:42.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:42 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:42.129+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:42.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:42 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:42.129+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:42.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:42 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:42.129+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:42.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:42 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:42.129+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:14.702081+0000 front 2024-09-11T08:33:14.702037+0000 (oldest deadline 2024-09-11T08:33:40.001909+0000) 2024-09-11T08:33:43.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:43.152+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:43.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:43.152+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:43.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:43.152+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:43.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:43 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:43.152+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:14.702081+0000 front 2024-09-11T08:33:14.702037+0000 (oldest deadline 2024-09-11T08:33:40.001909+0000) 2024-09-11T08:33:44.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:44 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:44.180+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:44.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:44 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:44.180+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:44.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:44 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:44.180+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:44.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:44 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:44.180+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:14.702081+0000 front 2024-09-11T08:33:14.702037+0000 (oldest deadline 2024-09-11T08:33:40.001909+0000) 2024-09-11T08:33:45.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:45 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:45.144+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:45.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:45 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:45.144+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:45.571 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:45 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:45.144+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:45.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:45 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:45.144+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:14.702081+0000 front 2024-09-11T08:33:14.702037+0000 (oldest deadline 2024-09-11T08:33:40.001909+0000) 2024-09-11T08:33:45.572 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:45 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:45.144+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.139:6802 osd.4 since back 2024-09-11T08:33:20.002372+0000 front 2024-09-11T08:33:20.002459+0000 (oldest deadline 2024-09-11T08:33:44.702323+0000) 2024-09-11T08:33:46.568 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:46.115+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6803 osd.0 since back 2024-09-11T08:32:51.598396+0000 front 2024-09-11T08:32:51.598265+0000 (oldest deadline 2024-09-11T08:33:17.498212+0000) 2024-09-11T08:33:46.568 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:46.115+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6807 osd.1 since back 2024-09-11T08:32:57.999017+0000 front 2024-09-11T08:32:57.999066+0000 (oldest deadline 2024-09-11T08:33:23.899025+0000) 2024-09-11T08:33:46.568 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:46.115+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6811 osd.2 since back 2024-09-11T08:33:07.101106+0000 front 2024-09-11T08:33:07.101042+0000 (oldest deadline 2024-09-11T08:33:31.200941+0000) 2024-09-11T08:33:46.568 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:46.115+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.110:6815 osd.3 since back 2024-09-11T08:33:14.702081+0000 front 2024-09-11T08:33:14.702037+0000 (oldest deadline 2024-09-11T08:33:40.001909+0000) 2024-09-11T08:33:46.569 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7[54879]: 2024-09-11T08:33:46.115+0000 7f9b7cfb7640 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.139:6802 osd.4 since back 2024-09-11T08:33:20.002372+0000 front 2024-09-11T08:33:20.002459+0000 (oldest deadline 2024-09-11T08:33:44.702323+0000) 2024-09-11T08:33:46.821 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 podman[70512]: 2024-09-11 08:33:46.568738817 +0000 UTC m=+5.451260066 container died 682cfe9f2d843e900b40bae01646e40a7493d104d59e3ef9893df3c449cf5315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:47.168 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 podman[70512]: 2024-09-11 08:33:46.926610688 +0000 UTC m=+5.809131939 container cleanup 682cfe9f2d843e900b40bae01646e40a7493d104d59e3ef9893df3c449cf5315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-11T08:33:47.168 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:46 smithi139 bash[70512]: ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7 2024-09-11T08:33:47.475 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:47 smithi139 podman[70527]: 2024-09-11 08:33:47.169139302 +0000 UTC m=+0.592081550 container remove 682cfe9f2d843e900b40bae01646e40a7493d104d59e3ef9893df3c449cf5315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-11T08:33:47.763 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:47 smithi139 podman[70619]: 2024-09-11 08:33:47.378517262 +0000 UTC m=+0.015963537 image pull 4a918172a49766385959227c53e75e2c302e923539324792cb5bee6cad0b2f2b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40 2024-09-11T08:33:47.763 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:47 smithi139 podman[70619]: 2024-09-11 08:33:47.4868138 +0000 UTC m=+0.124260062 container create 118850fc37b8f9dd9d5f4bdbabd812234e92626376b98fb4073af38f3ae932d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:47.763 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:47 smithi139 podman[70619]: 2024-09-11 08:33:47.697231393 +0000 UTC m=+0.334677656 container init 118850fc37b8f9dd9d5f4bdbabd812234e92626376b98fb4073af38f3ae932d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828) 2024-09-11T08:33:47.763 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:47 smithi139 podman[70619]: 2024-09-11 08:33:47.70052033 +0000 UTC m=+0.337966598 container start 118850fc37b8f9dd9d5f4bdbabd812234e92626376b98fb4073af38f3ae932d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2) 2024-09-11T08:33:47.763 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:47 smithi139 podman[70619]: 2024-09-11 08:33:47.763786004 +0000 UTC m=+0.401232278 container attach 118850fc37b8f9dd9d5f4bdbabd812234e92626376b98fb4073af38f3ae932d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:33:48.071 INFO:journalctl@ceph.osd.7.smithi139.stdout:Sep 11 08:33:47 smithi139 podman[70619]: 2024-09-11 08:33:47.882428316 +0000 UTC m=+0.519874593 container died 118850fc37b8f9dd9d5f4bdbabd812234e92626376b98fb4073af38f3ae932d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:33:48.590 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@osd.7.service' 2024-09-11T08:33:48.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:48.629 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-11T08:33:48.629 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-11T08:33:48.629 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@rgw.foo.a 2024-09-11T08:33:48.948 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:33:48 smithi110 systemd[1]: Stopping Ceph rgw.foo.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:33:49.403 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:33:48 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-rgw-foo-a[65868]: 2024-09-11T08:33:48.948+0000 7fb42f7fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-11T08:33:49.403 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:33:48 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-rgw-foo-a[65868]: 2024-09-11T08:33:48.948+0000 7fb4380947c0 -1 shutting down 2024-09-11T08:33:51.903 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:33:51 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:33:51.628Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi110.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.110:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-09-11T08:33:51.903 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:33:51 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:33:51.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi110.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.110:8443: connect: connection refused" 2024-09-11T08:33:51.903 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:33:51 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:33:51.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-09-11T08:33:59.265 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:33:58 smithi110 bash[83268]: time="2024-09-11T08:33:58Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2850e098-7016-11ef-bcea-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-11T08:33:59.265 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:33:58 smithi110 podman[83268]: 2024-09-11 08:33:58.969195978 +0000 UTC m=+10.135418318 container died e647fd44b0b65fedac5a2c94edcb1d2e9c4e079df562a38bc4afd65759c0c965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-11T08:33:59.589 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:33:59 smithi110 podman[83268]: 2024-09-11 08:33:59.339230226 +0000 UTC m=+10.505452560 container cleanup e647fd44b0b65fedac5a2c94edcb1d2e9c4e079df562a38bc4afd65759c0c965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-11T08:33:59.589 INFO:journalctl@ceph.rgw.foo.a.smithi110.stdout:Sep 11 08:33:59 smithi110 bash[83268]: ceph-2850e098-7016-11ef-bcea-c7b262605968-rgw-foo-a 2024-09-11T08:33:59.723 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@rgw.foo.a.service' 2024-09-11T08:33:59.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:33:59.758 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-11T08:33:59.758 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-11T08:33:59.758 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-2850e098-7016-11ef-bcea-c7b262605968@prometheus.a 2024-09-11T08:34:00.068 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:33:59 smithi139 systemd[1]: Stopping Ceph prometheus.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:34:00.321 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:34:00.321 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-11T08:34:00.321 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.068Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.069Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.069Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.069Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a[64410]: ts=2024-09-11T08:34:00.069Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-11T08:34:00.322 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 podman[70724]: 2024-09-11 08:34:00.078753309 +0000 UTC m=+0.120090317 container died c90a41240ab010691965638bf12a4ff52f6c8fa68286ae631017b16d600577bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:00.735 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 podman[70724]: 2024-09-11 08:34:00.426177013 +0000 UTC m=+0.467514011 container cleanup c90a41240ab010691965638bf12a4ff52f6c8fa68286ae631017b16d600577bd (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:00.735 INFO:journalctl@ceph.prometheus.a.smithi139.stdout:Sep 11 08:34:00 smithi139 bash[70724]: ceph-2850e098-7016-11ef-bcea-c7b262605968-prometheus-a 2024-09-11T08:34:00.897 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2850e098-7016-11ef-bcea-c7b262605968@prometheus.a.service' 2024-09-11T08:34:00.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-11T08:34:00.934 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-11T08:34:00.934 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2850e098-7016-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-11T08:34:01.833 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:34:01.629Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi110.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.110:8443: connect: connection refused" 2024-09-11T08:34:01.833 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:34:01.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi110.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.110:8443: connect: connection refused" 2024-09-11T08:34:01.833 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:01 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:34:01.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi139.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.139:8443: connect: connection refused" 2024-09-11T08:34:02.894 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:02 smithi110 systemd[1]: Stopping Ceph alertmanager.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:34:03.153 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:02 smithi110 ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a[74237]: ts=2024-09-11T08:34:02.894Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-11T08:34:03.153 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:02 smithi110 podman[83710]: 2024-09-11 08:34:02.915215153 +0000 UTC m=+0.113474392 container died a82b469208e5672a099e926d90285602278f5d0a26e4502bcf3c6c5fad2054ba (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:03.481 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:03 smithi110 podman[83710]: 2024-09-11 08:34:03.255670932 +0000 UTC m=+0.453930187 container cleanup a82b469208e5672a099e926d90285602278f5d0a26e4502bcf3c6c5fad2054ba (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:03.482 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:03 smithi110 bash[83710]: ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a 2024-09-11T08:34:03.482 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:03 smithi110 podman[83722]: 2024-09-11 08:34:03.39839537 +0000 UTC m=+0.480310706 container remove a82b469208e5672a099e926d90285602278f5d0a26e4502bcf3c6c5fad2054ba (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:03.482 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:03 smithi110 podman[83722]: 2024-09-11 08:34:03.482108928 +0000 UTC m=+0.564024263 volume remove 371cb645370386f5ad8b84407722bafa30e6c6ece930800c7b93b0f32f9a4ffa 2024-09-11T08:34:03.886 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:03 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-11T08:34:03.886 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:03 smithi110 systemd[1]: Stopped Ceph alertmanager.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:34:03.886 INFO:journalctl@ceph.alertmanager.a.smithi110.stdout:Sep 11 08:34:03 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@alertmanager.a.service: Consumed 1.129s CPU time. 2024-09-11T08:34:04.136 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:03 smithi110 systemd[1]: Stopping Ceph node-exporter.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:34:04.392 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 podman[83829]: 2024-09-11 08:34:04.137091816 +0000 UTC m=+0.114460917 container died 34f1184be6c7ccab5ec2d614a43f71b8d95e40fc80d812eb7e535266cbf039ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:04.652 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 podman[83829]: 2024-09-11 08:34:04.484651872 +0000 UTC m=+0.462020978 container cleanup 34f1184be6c7ccab5ec2d614a43f71b8d95e40fc80d812eb7e535266cbf039ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:04.653 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 bash[83829]: ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a 2024-09-11T08:34:05.114 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 podman[83841]: 2024-09-11 08:34:04.735897562 +0000 UTC m=+0.602646798 container remove 34f1184be6c7ccab5ec2d614a43f71b8d95e40fc80d812eb7e535266cbf039ef (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-11T08:34:05.114 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:34:05.114 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-11T08:34:05.114 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 systemd[1]: Stopped Ceph node-exporter.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:34:05.114 INFO:journalctl@ceph.node-exporter.a.smithi110.stdout:Sep 11 08:34:04 smithi110 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.a.service: Consumed 2.411s CPU time. 2024-09-11T08:34:05.527 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2850e098-7016-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-11T08:34:07.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:06 smithi139 systemd[1]: Stopping Ceph iscsi.iscsi.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:34:07.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:07 smithi139 ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a[59596]: debug Shutdown received 2024-09-11T08:34:17.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:17 smithi139 bash[71109]: time="2024-09-11T08:34:17Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-11T08:34:17.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:17 smithi139 podman[71109]: 2024-09-11 08:34:17.154767356 +0000 UTC m=+10.127043845 container died 172b96c6173ad30847171680d413d3c4bfd763c838e11a34813cba7dcf4d08f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-11T08:34:17.779 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:17 smithi139 podman[71109]: 2024-09-11 08:34:17.503727508 +0000 UTC m=+10.476003986 container cleanup 172b96c6173ad30847171680d413d3c4bfd763c838e11a34813cba7dcf4d08f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-11T08:34:17.779 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:17 smithi139 bash[71109]: ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a 2024-09-11T08:34:18.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:17 smithi139 podman[71122]: 2024-09-11 08:34:17.779909682 +0000 UTC m=+0.624381639 container remove 172b96c6173ad30847171680d413d3c4bfd763c838e11a34813cba7dcf4d08f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-11T08:34:18.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:17 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-11T08:34:18.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 podman[71166]: 2024-09-11 08:34:18.034640307 +0000 UTC m=+0.111405704 container died 6d05d21df435000f4b718a1b8bfccd8c013de6e2c34ebb49d114f40eefc16b2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-09-11T08:34:18.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 podman[71166]: 2024-09-11 08:34:18.349054455 +0000 UTC m=+0.425819848 container cleanup 6d05d21df435000f4b718a1b8bfccd8c013de6e2c34ebb49d114f40eefc16b2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-11T08:34:18.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 bash[71166]: ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-11T08:34:18.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-11T08:34:18.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 59442 (conmon) remains running after unit stopped. 2024-09-11T08:34:18.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 71178 (podman) remains running after unit stopped. 2024-09-11T08:34:18.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 systemd[1]: Stopped Ceph iscsi.iscsi.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:34:18.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Consumed 2.640s CPU time. 2024-09-11T08:34:18.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi139.stdout:Sep 11 08:34:18 smithi139 podman[71178]: 2024-09-11 08:34:18.565703564 +0000 UTC m=+0.535075986 container remove 6d05d21df435000f4b718a1b8bfccd8c013de6e2c34ebb49d114f40eefc16b2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ef9c65e2a6ea9377a732e309273ea235301c4d330f09e2972aeebc38173d3e40, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-10-2008-quincy-d0e6828, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-11T08:34:19.071 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:18 smithi139 systemd[1]: Stopping Ceph grafana.a for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:34:19.072 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:18 smithi139 podman[71299]: 2024-09-11 08:34:18.980747872 +0000 UTC m=+0.110877608 container died 932633c9d73ca13d27e7d7baa87b5035b7c3950aeaa57c0aa456ff887a5b7fa2 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.component=ubi8-container, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-09-11T08:34:19.564 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:19 smithi139 podman[71299]: 2024-09-11 08:34:19.294496596 +0000 UTC m=+0.424626332 container cleanup 932633c9d73ca13d27e7d7baa87b5035b7c3950aeaa57c0aa456ff887a5b7fa2 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, name=ubi8, version=8.5, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-11T08:34:19.564 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:19 smithi139 bash[71299]: ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a 2024-09-11T08:34:19.821 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:19 smithi139 podman[71311]: 2024-09-11 08:34:19.56443566 +0000 UTC m=+0.585263972 container remove 932633c9d73ca13d27e7d7baa87b5035b7c3950aeaa57c0aa456ff887a5b7fa2 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-grafana-a, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, io.buildah.version=1.32.0, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, maintainer=Paul Cuzner , release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-09-11T08:34:19.821 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:19 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:34:19.821 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:19 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-11T08:34:19.821 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:19 smithi139 systemd[1]: Stopped Ceph grafana.a for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:34:19.821 INFO:journalctl@ceph.grafana.a.smithi139.stdout:Sep 11 08:34:19 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@grafana.a.service: Consumed 6.845s CPU time. 2024-09-11T08:34:20.321 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:19 smithi139 systemd[1]: Stopping Ceph node-exporter.b for 2850e098-7016-11ef-bcea-c7b262605968... 2024-09-11T08:34:20.321 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 podman[71422]: 2024-09-11 08:34:20.201039615 +0000 UTC m=+0.095270819 container died 65651de3087bcecd082b8fafefdd42405c9a414afc507c39cfa0d4f3d4edf4fc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:34:20.700 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 podman[71422]: 2024-09-11 08:34:20.498069574 +0000 UTC m=+0.392300768 container cleanup 65651de3087bcecd082b8fafefdd42405c9a414afc507c39cfa0d4f3d4edf4fc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:34:20.701 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 bash[71422]: ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b 2024-09-11T08:34:21.030 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 podman[71435]: 2024-09-11 08:34:20.701449836 +0000 UTC m=+0.504385447 container remove 65651de3087bcecd082b8fafefdd42405c9a414afc507c39cfa0d4f3d4edf4fc (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2850e098-7016-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-11T08:34:21.031 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-11T08:34:21.031 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-11T08:34:21.031 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 systemd[1]: Stopped Ceph node-exporter.b for 2850e098-7016-11ef-bcea-c7b262605968. 2024-09-11T08:34:21.031 INFO:journalctl@ceph.node-exporter.b.smithi139.stdout:Sep 11 08:34:20 smithi139 systemd[1]: ceph-2850e098-7016-11ef-bcea-c7b262605968@node-exporter.b.service: Consumed 2.429s CPU time. 2024-09-11T08:34:21.463 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:34:21.497 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-11T08:34:21.525 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-11T08:34:21.527 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274/remote/smithi110/crash 2024-09-11T08:34:21.528 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/crash -- . 2024-09-11T08:34:21.645 INFO:teuthology.orchestra.run.smithi110.stderr:tar: /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-11T08:34:21.645 INFO:teuthology.orchestra.run.smithi110.stderr:tar: Error is not recoverable: exiting now 2024-09-11T08:34:21.647 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274/remote/smithi139/crash 2024-09-11T08:34:21.648 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/crash -- . 2024-09-11T08:34:21.681 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/2850e098-7016-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-11T08:34:21.681 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2024-09-11T08:34:21.682 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-11T08:34:21.682 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-11T08:34:21.710 INFO:teuthology.orchestra.run.smithi110.stderr:grep: /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph.log: No such file or directory 2024-09-11T08:34:21.712 INFO:tasks.cephadm:Compressing logs... 2024-09-11T08:34:21.712 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T08:34:21.756 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T08:34:21.781 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/tcmu-runner.log 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:: No such file or directory 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr: 63.5% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/tcmu-runner.log.gz 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-11T08:34:21.782 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-11T08:34:21.784 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-11T08:34:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-11T08:34:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:: No such file or directory 2024-09-11T08:34:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-11T08:34:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-11T08:34:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-11T08:34:21.785 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-11T08:34:21.786 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 92.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-11T08:34:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-11T08:34:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.c.log: /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-11T08:34:21.786 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-11T08:34:21.789 INFO:teuthology.orchestra.run.smithi139.stderr: 91.2% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-11T08:34:21.789 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-11T08:34:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log: 58.7% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-11T08:34:21.790 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-11T08:34:21.794 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.3.log: 91.2% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-11T08:34:21.798 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-11T08:34:21.808 INFO:teuthology.orchestra.run.smithi139.stderr: 88.8% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-11T08:34:21.992 INFO:teuthology.orchestra.run.smithi110.stderr: 88.8% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-11T08:34:22.021 INFO:teuthology.orchestra.run.smithi139.stderr: 92.2% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-11T08:34:22.077 INFO:teuthology.orchestra.run.smithi110.stderr: 92.0% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-11T08:34:22.084 INFO:teuthology.orchestra.run.smithi139.stderr: 92.9% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-11T08:34:22.107 INFO:teuthology.orchestra.run.smithi139.stderr: 93.0% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-11T08:34:22.141 INFO:teuthology.orchestra.run.smithi139.stderr: 93.2% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-11T08:34:22.194 INFO:teuthology.orchestra.run.smithi110.stderr: 92.8% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-11T08:34:22.210 INFO:teuthology.orchestra.run.smithi110.stderr: 93.0% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-11T08:34:22.215 INFO:teuthology.orchestra.run.smithi139.stderr: 93.2% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-11T08:34:22.217 INFO:teuthology.orchestra.run.smithi139.stderr: 2024-09-11T08:34:22.217 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m0.447s 2024-09-11T08:34:22.217 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m1.651s 2024-09-11T08:34:22.217 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m0.071s 2024-09-11T08:34:22.279 INFO:teuthology.orchestra.run.smithi110.stderr: 93.1% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-11T08:34:22.326 INFO:teuthology.orchestra.run.smithi110.stderr: 93.2% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-11T08:34:22.397 INFO:teuthology.orchestra.run.smithi110.stderr: 91.1% -- replaced with /var/log/ceph/2850e098-7016-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-11T08:34:22.399 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-11T08:34:22.399 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m0.630s 2024-09-11T08:34:22.399 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m2.906s 2024-09-11T08:34:22.399 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.123s 2024-09-11T08:34:22.400 INFO:tasks.cephadm:Archiving logs... 2024-09-11T08:34:22.400 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274/remote/smithi110/log 2024-09-11T08:34:22.401 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T08:34:22.836 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274/remote/smithi139/log 2024-09-11T08:34:22.837 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T08:34:23.104 INFO:tasks.cephadm:Removing cluster... 2024-09-11T08:34:23.104 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2850e098-7016-11ef-bcea-c7b262605968 --force 2024-09-11T08:34:23.610 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2850e098-7016-11ef-bcea-c7b262605968 --force 2024-09-11T08:34:24.088 INFO:tasks.cephadm:Removing cephadm ... 2024-09-11T08:34:24.088 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-11T08:34:24.106 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-11T08:34:24.121 INFO:tasks.cephadm:Teardown complete 2024-09-11T08:34:24.121 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T08:34:24.132 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T08:34:24.132 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T08:34:24.148 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T08:34:24.240 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T08:34:24.249 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T08:34:24.249 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:34:24.252 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:34:24.264 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-09-11T08:34:24.266 INFO:teuthology.orchestra.run.smithi139.stderr:bash: line 1: ntpq: command not found 2024-09-11T08:34:24.268 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T08:34:24.268 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-09-11T08:34:24.268 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 -1762us[-1776us] +/- 60ms 2024-09-11T08:34:24.269 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 36 +1347us[+1334us] +/- 66ms 2024-09-11T08:34:24.269 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +771us[ +757us] +/- 81ms 2024-09-11T08:34:24.269 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:34:24.270 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T08:34:24.270 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2024-09-11T08:34:24.270 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 -1763us[-1768us] +/- 60ms 2024-09-11T08:34:24.270 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 +1292us[+1287us] +/- 66ms 2024-09-11T08:34:24.270 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +761us[ +761us] +/- 81ms 2024-09-11T08:34:24.271 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T08:34:24.271 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T08:34:24.280 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T08:34:24.281 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T08:34:24.371 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T08:34:24.444 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T08:34:24.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T08:34:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:34:24.542 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T08:34:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:34:24.570 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T08:34:24.580 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T08:34:24.616 INFO:teuthology.task.internal:Duration was 1434.248650 seconds 2024-09-11T08:34:24.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T08:34:24.649 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T08:34:24.649 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T08:34:24.652 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T08:34:24.688 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T08:34:24.690 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T08:34:24.942 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T08:34:24.942 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-09-11T08:34:24.943 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T08:34:25.005 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-09-11T08:34:25.006 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T08:34:25.047 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T08:34:25.047 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T08:34:25.049 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T08:34:25.114 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T08:34:25.114 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T08:34:25.117 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T08:34:25.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T08:34:25.412 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T08:34:25.413 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T08:34:25.437 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T08:34:25.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T08:34:25.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:34:25.480 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:34:25.501 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-09-11T08:34:25.525 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-09-11T08:34:25.549 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:34:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:34:25.574 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T08:34:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:34:25.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T08:34:25.618 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T08:34:25.619 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274/remote/smithi110 2024-09-11T08:34:25.619 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T08:34:25.658 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-11_06:53:43-orch:cephadm-wip-guits-quincy-2024-09-10-2008-quincy-distro-default-smithi/7901274/remote/smithi139 2024-09-11T08:34:25.658 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T08:34:25.694 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T08:34:25.694 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T08:34:25.697 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T08:34:25.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T08:34:25.753 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T08:34:25.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T08:34:25.783 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T08:34:25.783 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T08:34:25.785 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T08:34:25.798 INFO:teuthology.orchestra.run.smithi110.stdout: 265602 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 08:34 /home/ubuntu/cephtest 2024-09-11T08:34:25.800 INFO:teuthology.orchestra.run.smithi139.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 08:34 /home/ubuntu/cephtest 2024-09-11T08:34:25.801 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T08:34:25.816 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T08:34:25.887 INFO:teuthology.run:Summary data: description: orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} duration: 1434.2486498355865 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-11T08:34:25.887 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T08:34:25.967 INFO:teuthology.run:pass